Ruby, Javascript, Erlang blog: Snakes, Gems & Coffee

DRAFT: Why proponents of marriage equality should love graph databases, Part 2: A Reply to 'The Database Engineering Perspective'

2010-04-03T11:37:00.001-05:00

So in part 1 I provided an overview of problems relational databases (RDBMS) have with modelling relationships as well as entities that have datasets that are only partially structured in structure and definition based on my experiences.

This post will:

introduce new concepts utilized in graph databases and specific terminology for Neo databases
review the most flexible schemas that accommodate both same-sex and opposite-sex marriages alike that were presented in the blog post I am responding to, 'Gay Marriage: The Database Engineering Perspective'.
describe how graph databases could overcome the relational database shortcomings
finally I will present a snippet of code (not meant for production use, but merely to demonstrate how we can use Neo4J) that makes it more natural to represent richer relationships between entities (or nodes as Neo4J calls them) that have substance to them (i.e. attributes in this case)

New Concepts in Graph Databases

Before I can explain how graph databases on a conceptual level can overcome most if not all (for most of the time anyway) of the relational databases (RDBMS) shortcomings mentioned in the previous post, we need to be introduced to new concepts and terminology.
There are two basic classes of objects in a graph database, they are:

Nodes: a node is basically an entity as RDBMS people (like probably you and I) are familiar with. A node could represent a person, customer, blog post, photograph, video or tweet. It isn't always true, nor is it a good idea to think of graph database concepts only in terms of RDBMS concepts, but we could consider most tables that do not represent an actual relationship or association between other tables as an entity. It is a simplification, so use only in this initial learning phase cautiously. A node can have zero or more properties.
Relatioships: a relationship is the concept of an association between two nodes that needs to be represented in your data store somehow. A relationship can also have zero or more properties, just like a node.

The third concept in graph databases is that of a property. It unifies nodes and relationships as both first class citizens of the data store, which is quite unlike relational databases.

In a relational database we try to either reduce an association between two entities/tables to a small handful of cases, which might use references (if supported by that particular vendor and version of RDBMS), create "join tables" or we might force a natural relationship to become a table itself so that we are able to capture pertinent attributes for it. There is no other way of handling relationships. In many cases it might not be an issue to reduce the problem in these ways, however, the more and more interconnected our entities become with new types of relationships it seems (to me) that this method of modelling data quickly becomes problematic.

The following are other useful terms that the Neo graph database family uses, but conceptually most of these will also have a synonym in other graph databases:

Traversers: in graph databases "querying" for specific data is not done via declarative query statements and clauses like SQL. Rather we define traversers that are composed of the following elements:
- the starting node
- the relationship types needing to be traversed
- the stop criteria
- the selection criteria
- traversing order (e.g. breadth first, etc.)
We need to know the starting node so the graph database knows where to start the traversal. The traverser will then only move along relationships of the types given by the second element of a traverser, evaluate the selection criteria to know which nodes are relevant tot he "query", then evaluate whether the stop criteria applies to know when traversing should stop.
Indexers: In Neo4J there are several indexing utilities backed by a Lucene backend that make it easy to index actual nodes, full text and based on timelines. This allows us to look up relevant starting nodes for traversing.

How can graph databases overcome relational database shortcomings?

Now we will have a quick peak at how the new concepts and ideas from graph databases can resolve the complaints I had about relational databases:

In response to A lot of real world data isn't highly structured (only partially): with graph databases we only need to set the properties a specific node actually has. We do not need to fill in lots of nulls in attributes that aren't relevant like we do in RDBMSes. Our graph database "model" interface might be able to add any constraints that are necessary. While, in situations where highly structured data sets really do occur, I prefer having two sets of constraints - those on the relational database level and those in the "model" (application tier) - however, it can quickly become hard to manage when two sets of constraints that should be identical are defined in vastly different ways (languages).
In response to Object to relational mapping (ORM) constraints and disjoints: using nodes and relationships conceptually on the graph database layer you have little if any translation between 'objects' that represent nodes or relationships.
In response to Weak and inefficient "traversal" support: this isn't a problem with graph databases. Traversing data is much more relevant when dealing with data that is naturally in a network or graph formation already. Relational databases cannot handle networks or graph-like data sets without a lot of workarounds.
In response to Maintaining and evolving relational schemas: when the data sets you are dealing with are not highly structured and densely populated evolving strict data schemas do not need to be maintained and with Neo4J's flexible attribute setting/getting, data structures can be flexibly evolved when the data itself changes, however, this raises another issue of application code being able to read the old attributes in the graph database, but I will talk about that in Part 3.

Gay marriage: the graph database solution

Why proponents of marriage equality should love graph databases, Part 1: A Reply to 'The Database Engineering Perspective'

2010-03-13T10:28:00.003-06:00

This is a response to the excellent Gay marriage: the database engineering perspective blog post from November 2008 by Sam Hughs. The only discussion that didn't happen in his blog post was offering a NOSQL alternative to the gay marriage database problem, but I will not hold it against Sam Hughs because it was a comprehensive look at relational database schema designs for modelling marriage!

This response is a little dabble (i.e. not as thorough as the original post) into how graph databases could model monogamous same sex marriages without much problem and also allow for polyamorous marriages quite naturally. My goal here is to demonstrate how graph databases provide a great amount of flexibility without very much work at all, especially when the data application is just as concerned with connections between entities (aka records in RDBMS or nodes in graph databases) than just the entities themselves.

First off I want to look at the inherent problems with modelling real world data in relational databases and how the approach of graph databases can overcome many, if not all, of these problems. Then I will launch into a specific snippet of code to demonstrate modelling marriages that can be between any two consenting adults rather than just one man and one woman. Then I hope to outline how this could be extended to model poly amorous (including polygamous) relationships without much more work. Near the end I will also include a section at the end that discusses shortcomings of the graph database approach as well for completeness. My discussion will be focused on using Neo4J as it is the only comprehensive graph database I am aware of that is open source.

Update (2010-03-13): Boris (in the comments) mentioned that there is another open source graph database called HyperGraphDB

Update (2010-04-03): Johannes (via email) told me about InfoGrid, which looks like a very interesting alternative to Neo4J with a slightly different way of doing things.

What is wrong with relational databases?

For the last 14 years I have been using, designing, maintaining and administering relational databases in some capacity as a software programmer, developer, engineer and now as an applications architect. During that time I have found the following problems with using relational databases:

A lot of real world data isn't highly structured (only partially): this is probably the biggest problem with using relational databases (for everything) the way they are supposed to be. Sure you can add a blob, clob or text field to a table and add an arbitrary structure of data that depends on the record, which can be parsed by the application, but that would be a violation of all things relational and you would be paying a heavy price for doing this in a relational database depending on how big these "blobs" generally are. Not to mention you miss out on query-ability, which is something relational databases do well on highly structured datasets with the relevant indexes defined and a decent attempt at normalization. Note: I think relational databases are a fine thing when utilized to model data that truly is highly structured in the wild where the entities, not the relationships matter the most.
Object to relational mapping (ORM) constraints and disjoints: For almost a decade I have been using ORM libraries such as TOPLink (Java), Persistence (C++), Hibernate (Java), ActiveRecord (Ruby), DataMapper (Ruby), SQLObject (Python) and others. They each had their own specific problems at times, but they also possessed a set of common problems that resulted in creating a far from seamless integration between the object oriented layer (that a vast number of business systems and web applications are currently written with today) and the distinct properties of a relational database. These common object to relational constraints and disjointedness is not an ORM tool issue, rather it is a problem of trying to shove a round pin into a square hole (paraphrasing a comment made in page 3 of 'The Neo Database: A Technology Introduction').
Weak and inefficient "traversal" support: When I first started writing rich domain models in the nineties (that is last century for all you whipper-snappers) the focus of design was on the actual entities not relationships between the entities. Sure occasionally you had to model a relationship as an "entity", but for the most part you could do your best to reduce your domain relationships as much as possible to a combination of contain a(n) (aka belongs_to in ActiveRecord), contained by (aka has_one in ActiveRecord), one-to-many (aka has_many in ActiveRecord) and/or many-to-many (aka has_and_belongs_to_many in ActiveRecord) as much as possible because you can't attach attributes to relationships in a relational database unless you attempt to make it an entity. If you are interested in deep and/or rich object graphs, however, you have to pay the penalty in any RDBMS with multiple joins, which are an expensive operation. Even with all the right indices defined and queries optimized you will find more than three levels of indirection (aka JOINs) will take a while on even a medium sized dataset. So as application developers, we are forced to add lazy loading and/or customized eager loading logic into our rich object domain layer for a number cases where performance matters, which in my view pollutes business logic and makes the code less maintainable going forward. This is far from ideal. It is also not ideal that data set traversal patterns change with new the introduction of new features, so often you will need to fork lazy loading logic in the business logic layer to satisfy you new requirements. Adding a lot more complexity to manage in the application tier.
Maintaining and evolving relational schemas: modifying relational database schemas has always been a little tricky at best. Today I take advantage of using ActiveRecord's "migrations" that loosely orders (by creation timestamp) a set of relational schema modifications to run (and we wrote a simple extension to wrap them in a transaction to save our sanity - why wasn't that the default to begin with? anyway...). Even though there is a method to the madness now, it is still a little crazy, especially when it comes time to run these migrations on the production server. I always miss a heart beat or two when I run the rake db:migrate RAILS_ENV=production command (yes, even after taking a snapshot).

It has been fun thinking out loud, but I have stuff to do today. Hey, I sometimes have a life, honest!:) So I have made an outline of what is to come in the next parts of this topic. I have already written some (bad) Java code to model same-sex and opposite-sex marriages consistently using a graph database (in this case Neo4J), but I plan on offering a Python snippet too, since I really can't stand the look of Java any more and the API doesn't make the case of Graph Databases for those coming from more terser feeling languages like Python, Ruby, Haskell, Erlang, Javascript (well if you use sane APIs like jQuery at least).

What is coming in Part 2?

How can graph databases overcome relational database shortcomings?
Gay marriage: the graph database solution

What is coming in Part 3?

Poly amorous marriage using a graph database
Graph databases: problems to watch out for

How agile practices improve code review

2010-01-30T21:27:00.015-06:00

On a mailing list recently someone posed a question about how to incorporate code reviews into an "agile methodology". While I had questions about the way the question was posed it got me thinking about how agile practices make code review happen more often, more organically and more transparently. Having different members of the team review code at different times during the development process on agile projects is both explicit, but also implicit.

Values

Let us back up a little here. First off agile is at its core a set of four ideals. Prefer:

individuals and interactions over processes and tools
working software over complete documentation
customer involvement over contract negotiation
adapting to changing needs over completing a stale plan

Principles/Tenets

From these ideals there are about 10 or so principles or tenets that are most commonly discussed among agile practitioners from most of the main camps. This is where it starts getting controversial as some agile camps acknowledge some of these principles and others do not necessarily, so here are just a few that I want to talk about in relation to increasing code review on agile projects:

Collective team ownership of product: meaning everyone on the team owns the product and nobody on the team is exclusive to just one or a couple of components of the product.
Embracing change: instead of continuing down a set path determined long ago that no longer applies, the team needs to adapt to customer's changing requirements at set (and regular) intervals (these might be called iteration planning meetings/negotiations, for example). This has various implications throughout the iteration/sprint.
Verify expectations continuously: meaning whenever possible, without impeding development, verify that customer expectations will be satisfied and no new additions have negative impacts.
Customers set priorities: instead of technical staff or outsourcing companies specifying technical priorities, customers specify business/feature priorities.

Practices

This is the point where we start seeing the translation into agile practices, many of while you will start to recognize:

Pair programming which arose from both collective team ownership and embracing change tenets.
Test-first practice came about mostly to satisfy embracing change, but also verify expectations continuously
Assigning new stories independently of previous ones. This means that no developer or pair has a niche component or product area and is used to satisfy the first tenet listed above.
Continuous integration that helps satisfy the verify expectations continously principle.
No fear refactoring, which is only advisable if your test/spec suite has good enough coverage,. This means you will likely revise poorly constructed code on agile projects vs non-agile projects more often and have the chance to do it more often too.

Obviously with the pair programming practice that some agile methods promote, you have another set of eyes reviewing code as it is written. This catches errors closer to the time it was written, which is a big time (and money) saver.

Applying a test-first approach means that you start out defining custom expectations of the code first before writing the actual code. This is reviewing, perhaps not code in the traditional way, but the requirements, which in my view is even more productive usually.

When a project manager assigns new stories to a developer or a pair without confining him/her/them to specific areas of the codebase, this increasing the transparency of the codebase and exposes hidden dragons to be exposed before too long and hopefully before being launched into production (but you never know). Again this increasing the number of eye balls on parts of the codebase.

With continuous integration you can incorporate metric tools to capture some metrics of the code so that parts of the code that are obviously in need to technical refactoring get highlighted earlier rather than later. Increasing transparency of the codebase, which is always a goal of code reviews.

Hopefully you can see where I am going with this and not think about code reviews in the traditional top-down way where every month or quarter a piece of code is reviewed from each developer in a formal way.

There are a number of different agile (and common) software practices that could be used, I can think of 4 more off the top of my head that improves codebase visibility across more of the team.

So Long, Farewell, Auf Wiedersehen, Goodbye...to Ruby & Rails (sort of)

2010-01-17T18:37:00.006-06:00

It's been obvious to many of my close geek comrades that for the last 18-22 months I have been getting more and more frustrated with the Ruby community and the "quality" of products in this space. The quality of people in the Ruby field has been tending towards the average PHP developer (PHP is a gateway drug for programming - i.e. not something to aspire to) and the recent Ruby mindset (as a whole though some dark corners still look interesting) has not been about doing things differently, but rather blending in with mainstream ideas. For some that is exciting as it means not fighting management just to get to use Ruby in the enterprise. I understand, I really do. Fighting management on technical platform is never fun.

I think it is safe to say the direction of the major libraries and frameworks that are prominent in the Ruby landscape will suffer from very similar conceptual limitations as the Java, J2EE and JEE frameworks that DHH once ridiculed (for good reason). Of course, this is all to be expected in the usage lifecycle of a programming language. In fact, I believe I predicted that I would get bored of Ruby in a blog post in 2006 within the next few years though it doesn't make this any easier.

Thanks mates!

I wish all friends in the Ruby and Rails worlds luck (I think you'll need it) and I thank (ex-)coworkers and project collaborators that have taught me something new for all the intellectual stimulation you have provided me. I am sad to be saying goodbye to those of you.

But...

I look forward to grazing new pastures. I will be working on my next generation financial products using the right tool for each primary job. Currently most of the Finsignia Platform is quite back-end and different pieces could take advantage of Haskell's concurrency and/or Erlang distribution. Yet there will inevitably need to be (at some point) rich web applications for UI, so (J)Ruby and Rails I might come knocking on your door again, but for now, adiós.

Cloud Computing, Part 2: Law, Privacy & Maturity

2009-09-29T21:59:00.007-05:00

At the CEO roundtable event I mentioned in my previous blog post about in Cloud Computing, Part 1 the other major business consideration that came up was how to protect your business' (and/or customers') assets properly. Ultimately anyone that wants to build a solid foundation for their business needs to tackle this well to be successful long-term. One person asked the speaker to elaborate about a recent case where a UK bank used servers in datacenters inside India where a datacenter employee stole customer information and used it to effectively steal money from these people's accounts for his own benefit. This could happen in any country, however, in this case this datacenter employee actually didn't break any laws inside of India (or broke one very minor law that had a very limited penalty). Ultimately this lack of relevant law and regulation in a foreign country is the major risk for a business using offshore datacenter resources. Unfortunately I didn't hear the name of the bank involved to be able to find a news article about this incident, however, the speaker had heard about this incident (it wasn't just a figment of the questioners imagination) and his response (I am paraphrasing here) was:

Countries with the most mature infrastructure, laws, regulations and standards regarding data and information law (as well as privacy) will probably be the places that larger firms will invest in for sensitive and mission critical systems of their organization despite the likely higher cost of building.

As you determine whether you are going to be a cloud consumer or provider in the future, make sure to keep an eye on these risk factors as some executives seem to only focus on the raw bottom line without conducting a thorough risk analysis to see if the cost savings are really worth the higher risks. Choose the best cloud computing partners for all your cost trimming AND risk mitigation needs.

Cloud Computing, Part 1: Delivery Models

2009-09-11T21:10:00.011-05:00

Today I went to a regional CEO Roundtable lunch talk given by Professor Roy Campbell at UIUC on Cloud Computing and why executives of any kind of business should care.

While I am not new to the concepts of cloud computing and multi-tenancy SaaS, since I have consumed various cloud services and designed multi-tenant architectures for two previous clients. However, there was one particular "overview" slide that Professor Campbell showed that got me thinking a little. It was titled Delivery Models.

It was a text slide with bullets, but I think a diagram (or two) could have communicated to the higher-level audience more than most of the first half of his presentation could have (see below and let me know if it is more effective).

I especially liked Professor Campbell's categorization of the Delivery Models. Until this lunch time I had really only considered there to be two cloud delivery models: SaaS and PaaS. I wasn't making a distinction between Amazon services like S3 and EC2 and that of Google's AppEngine. On reflection, I am glad I took two hours out of my busy Friday if only to consider the distinction made in this talk.

In a nutshell the slide has three major bullets:

Software as a Service
Platform as a Service
Infrastructure as a Service

Nice! Google AppEngine, essentially being an application container (that supports various frameworks now), is a platform, whereas S3 and EC2 provide infrastructure on demand.

Now unfortunately this is where Campbell left it. For those of us not unfamiliar with cloud computing, we can see the importance of this distinction so that we can develop a more solid business plan (for those of us that might be interested in using the cloud to generate revenue from). It may also help technically savvy people that aren't familiar with the cloud.

Hope this helps people make a choice between container services like AppEngine or Heroku (platform) and those such as S3 and EC2 (infrastructure) from a consumer side and perhaps those devising business models to focus on their core competencies.

Next time I want to talk about privacy, legal and compliance issues surrounding Cloud Computing as well as how business should weigh the risk of outsourcing their data handling (however that happens) to countries will less mature laws, regulation and oversight. Perhaps cloud governance would be the most appropriate term?

Alan Turing Can Now Rest In Peace

2009-09-10T21:42:00.007-05:00

While I am by no means the biggest fan of the New Labour government in my homeland (mostly due to Blair's sociopathic reign), I have to give Gordon Brown props (or at least his aids) for giving credit to Alan Turing and saying Britain, as a country, fucked up big time (sorry, I really couldn't think of another phrase that matched the intensity - I'll replace it if you suggest a good alternative).

You can read the Downing Street blog post here:
Treatment of Alan Turing was “appalling” - PM

Thanks from this particular graduate from University of the Manchester's Mathematics department who now works as a software engineer who also happens to be gay!

"Ta luv", as the Manc locals would say.

Faker, Ruby on Rails Snippet & Generator

2009-09-09T19:07:00.012-05:00

Hey blog readers (if there are any left), I know it has been a while, and I know I have left you high and dry, so today I decided I will share some code and software development thoughts with you. I am moving to a more snippet oriented blog posting style from now on. Previously I preferred long ramblings about the political ideology of programming languages and even how Ann Coulter was _more_ comprehensible when I used my Metafusion Crypto library on her words (actually that was a marketing lie - she sounded just as incomprehensible using Metafusion Crypto as she did when not using it). Of course, this was all in 2006-2007 when blogging was all about spewing small tidbits that taught the reader nothing, so I had to be different and attempt to find a reason for things. Now things are swinging so I am trying to do the opposite from the norm again. Ok, on with it. Over the last week I have been using the Rails 2.3.4 "seed" mechanism, which in essence attempts to provide 80% of seed-fu's features inside Rails core. In addition on my personal projects I have been using machinist as my factory framework of choice (although I am forced to use factory_girl for my clients), through which I was introduced to faker. A beautiful gem indeed. So to cut a long story short, I have found myself doing the following a hell of a lot in my brand new Rails 2.3.4 projects this week:


# in db/seeds.rb
def load_seed_files!
  seed_files = Dir["#{RAILS_ROOT}/db/seeds/all/*.rb"] + Dir["#{RAILS_ROOT}/db/seeds/#{RAILS_ENV}/*.rb"]
  seed_files.each do |file|
    puts "Loading seed file: #{file}"
    load(file)
  end
end

load_seed_files!
# When I add files to the db/seeds/all directory it seed ALL environments
# When I add files to the db/seeds/{development,staging,test,production} directories it seeds 
# for only that environment.

Note: I wouldn't recommend using seeds in just staging, test or production generally. If you need seed data in these environments you _probably_ need them in all environments OR (in the case of the test environment) you need to use a factory framework like machinist or factory_girl. Now where is faker? Well it's what I use inside these .rb files where faker comes into play:


# Faking a company name
>> Faker::Company.name
=> "King-Ruecker"
>> Faker::Company.name
=> "Cartwright, Prosacco and Smith"
>> Faker::Company.name
=> "Pagac, Adams and Barrows"
>> Faker::Company.name
=> "Satterfield and Sons"
>> Faker::Company.name
=> "Blick-Deckow"

# Faking a person's name
>> Faker::Name.first_name
=> "Ruthe"
>> Faker::Name.first_name
=> "Jameson"
>> Faker::Name.first_name
=> "Avery"
>> Faker::Name.first_name
=> "Jordan"
>> Faker::Name.first_name
=> "Ruthe"
>> Faker::Name.first_name
=> "Winona"
>> Faker::Name.last_name
=> "Brown"
>> Faker::Name.last_name
=> "Hayes"
>> Faker::Name.last_name
=> "Jerde"
>> Faker::Name.last_name
=> "Lindgren"
>> Faker::Name.last_name
=> "Considine"
>> Faker::Name.last_name
=> "Rodriguez"
>> Faker::Name.last_name
=> "Turner"
>> Faker::Name.name
=> "Mekhi Lebsack"
>> Faker::Name.name
=> "Esteban Brown"
>> Faker::Name.name
=> "Velma Crona"
>> Faker::Name.name
=> "Rollin Zemlak"
>> Faker::Name.name
=> "Casper Sanford"

# My favorite is coming up with BS (yes, BS is what you think it is):
>> Faker::Company.bs
=> "redefine ubiquitous e-commerce"
>> Faker::Company.bs
=> "synergize viral e-business"
>> Faker::Company.bs
=> "innovate real-time experiences"
>> Faker::Company.bs
=> "aggregate scalable systems"
>> Faker::Company.bs
=> "e-enable back-end functionalities"
>> Faker::Company.bs
=> "implement 24/365 models"

So the point of faker, if you haven't already figured out is that it generates data the way you would probably have in production so you are generating meaningful data in your development environment(s) where you want to show your client what the interface will look like with data inside. Now a word of caution: I would not advise you to use this in an end-user facing test environment. I sometimes call that staging or qa, but ultimately it is an environment where real end-users or clients test out the features in a stable environment (not development or continuous integration environment) so they can sign off on your stories. In these environments you should let the user create the data themselves using the admin forms or whatever UI you provide them because you want to see how they will break the system (and we all know they will somehow). It gives you a much better indication of how things will end up working in production, which is very important. Thanks for reading and let me know if this is yet again too rambly? How does it happen? I have the best intentions to just blurt out tidbits with no nuance, yet I always end up rambling on about things and give it a thought process (well sort of). Anyone want to be my editor to strip out all thought and insight so I can be just blog brainless "tip" blog posts like everyone else in the Rails community that wanted to be a micro celebrity (trust me I couldn't handle that kind of paparazzi attention and I don't even have a Macbook Pro to sling over my shoulder in a fashionable messenger bag)? I can't pay you anything, but you could share in my measely $0.63 that Google Adsense pays me each month (actually they don't actually pay until a threshold is met). You could buy one third of an MP3 from iTunes each month. Any takers? What an awesome deal! No really! PS Coming soon a simple generator and Rails application template (or applate as I like to call them) that will set you up with this so you don't have to copy and paste all the time. It'll be on GitHub.com/mbbx6spp. Ta ta. Update: As Damien in the comments pointed out indirectly, I really didn't do Faker justice. Please read the Faker RDocs for more information.

Design Patterns Book Club

2009-07-03T10:05:00.006-05:00

I just discovered that Hashrocket has a video book club where people can join in the conversation with a live videocast on USTREAM or watch older episodes on their own schedule. They are currently going through Russ Olson's Design Patterns in Ruby. This is a great idea, kudos to Hashrocket. I watched the latest episode on the Strategy pattern and had a couple of thoughts I would love Desi and the rest of Hashrocket to consider for subsequent episodes:

I liked that this specific episode wasn't very long (I know you were a little worried)
I think discussing anti-patterns that could be solved with this pattern you are discussing the episode is very valid (as the person sitting next to Desi said). Perhaps that could be explored more. Note: I have only seen that one episode so perhaps you do this in other episodes much more, but if you don't please consider it, because I feel that anti-patterns are very important to identify. Think of the design patterns as the treatment or medicine and the anti-patterns as the symptoms. Physicians need to be acutely aware of the symptoms and symptom combinations to be effective at prescribing the best course of treatment or medication. Therefore we should be teaching software engineers in the same way.
Also please consider putting the name of the design pattern under discussion in the title of the video for archival purposes.

Thanks goes to Hashrocket for the great idea and work of putting it together! It's about time the Ruby community starts to mature w.r.t. language-neutral concepts and these efforts will hopefully help those that came into software development from a less formal software engineering education path (I used to be one of them and had to learn from scratch myself) with a little more assistance. It was pretty lonely bookwork and experimental projects for me to learn this stuff when I first started out.

Google Wave is not the answer, XMPP is!

2009-05-29T13:00:00.013-05:00

Since yesterday afternoon (my time) all over the Twittersphere (and blogosphere) I kept hearing a ton of great reviews of the Google Wave API/protocol and how it will revolutionize communication and collaboration on the web. This morning I watched the Google Wave demonstration from yesterday and I have to say I am completely underwhelmed considering everyone else's reactions were ridiculously positive and had nothing measured or thoughtful to say about it. You can find my initial reaction of it, "Google Wave is just a simple XMPP extnesion", on my tumblelog. As someone that has had to write XMPP extensions for clients and my own internal apps, servers, API interfaces (mostly distributed monitoring of services, but a few more for media processing) I am saddened that anyone would think the current Google Wave protocol (which is just an XMPP extension) is extraordinarily ground breaking. There have been a number of existing extensions that tried to do VERY VERY similar things already and specific features of Google Wave were already exactly defined. What made the demonstration of Google Wave yesterday appear so impressive were the applications: the wave client and the web embedding. The responsiveness of the web application really made it stand out, NOT the underlying Google Wave protocol/API. For those who have never really gotten their hands dirty with XMPP in any real technical way, I expected they would not really know what it already provided in XMPP. Many "geeks" hyping up Google Wave seemed to think XMPP only supported instant messaging. That just happened to be the most popular and apparent XMPP extension, which many people call Jabber. XMPP itself is a lot more far reaching than just an instant messaging protocol and the fact that Google Wave was based on top of XMPP should tell you how flexible it is. Yes that is right, Google may not have really made a big deal about it (why would they if they can get extra credit when it isn't due), but Google Wave is just an XMPP extension and compared to other extensions I have had to interface with, not a particularly complicated one. Sure the applications that Google demonstrated as part of the Google Wave presentation were nifty for personal communication, but are they really ground breaking, multipurpose, or ubiquitous? No, it is only as ground breaking, multipurpose and ubiquitous as it's applications make it. Let us all now return to earth for a moment and praise Google for the things they do really well like web applications, but let us not pretend that Google is the mover and shaker in terms of the visionary of future underlying protocols. They still advocate developing HTTP applications for the future. HyperText Transfer Protocol (HTTP) is virtually on every digital device made by man today, but does it really lend itself to providing the distributed service oriented architecture we should be striving for and the protocol of the future? REST is not about the HTTP protocol. RESTful APIs can be creating in a number of ways and through a lot of different protocols. My take is (from a technical level) XMPP ought to be the protocol of the future, NOT XMPP on just the back-end and then adding tons of bloatware in between the client and the service just so that we can create an API or UI on top of an antiquated protocol (HTTP) that has seen better days. Anyone want to create a truly modular XMPP "browser" for the new "web" with me? Email me at susanpotter dot net if you want to join the future. UPDATE: Please subscribe to the "Join the Future" Google Group if you want to get involved in the XMPP browser project.

Erlang doesn't support currying? News to me!

2009-05-20T16:13:00.004-05:00

I keep seeing developers who only glance at Erlang and think that currying isn't supported. Nothing could be further from the truth...unless you are defining currying support only in terms of specific syntactic sugar in a language for currying. So let us step back and define what currying support really means:

Currying is the process of transforming a function that takes multiple arguments into a function that takes just a single argument and returns another function if any arguments are still needed. http://www.haskell.org/haskellwiki/Currying

Let us look at a simple (but hopefully not too contrived, albeit it _slightly_ contrived) example:


%% currying_example.erl
-module(currying_example).

-compile([export_all]).

multiply(X, Y) -> X*Y.
doubler() -> fun(X) -> multiply(2, X) end.

In your erl shell try the following:


1> c(currying_example).
{ok,currying_example}
2> D = currying_example:doubler().
#Fun
3> D(6).
12
4> q().
ok

As you can see in the module code above, Erlang supports what is called anonymous functions (e.g. fun (X) -> somePredefineFunction(OtherVarOrConst, X) end), which then allows us to manufacture single argument functions in a way that is clear and expressive in the code IMO. For those that haven't been following the functional programming debates over the last week or two around the blogosphere, please see the following blog posts (some are pretty funny and will make you smile at least twice with their intended absurdity):

Previous posts on this blog that are related are:

I will try to put together a few snippets on currying support in Python, Javascript and Ruby tonight after my Greek class (which I am late for now) if I have time as a quick exercise:)

Twitter4R 0.3.1 and the future

2009-05-18T12:16:00.005-05:00

Hi Twitter4R users, I know I have been rather absent in terms of my Twitter4R contributions and ongoing development, but despite my absence I have _finally_ officially released 0.3.1 with a lot more new features and a few bug fixes more than I initially expected to release 0.3.1 with. Some of the goodies included in the official 0.3.1 release are:

Added raw Twitter Search API support. Meaning you can use any of the HTTP parameters the Twitter HTTP-based Search API will expect. E.g. twitter.search(:to => "twitter4r", :rpp => 10) or twitter.search(:q => "twitter4r", :rpp => 15, :geocode => "40.112186,-88.207592,250km")
Profile, theme API coverage on Twitter REST API.
Social Graphing API. E.g. twitter.graph(:friends, user)
Added "reply" support at all levels (low-level to model-level)
Bugfix: the infamous URI.encode to CGI.escape change.
Rate limit information (still needs to be able to support the HTTP header method, which I actually suggested to Twitter developers some time back, but they only recently added it when my bandwidth for OSS contribution was minimal) :(
much more

Now the future of the Twitter4R project. I have made a LOT of mistakes with this Twitter4R OSS project venture, so I am going to try to fix a bunch of these things:

Create a new set of Git repos on GitHub that separates the education portions (screencast, etc.) from the core source code of the project so that people do not need to download ridiculous amounts of data to their local drive just to fork Twitter4R on GitHub as it required at present. Announcements about this will be coming soon!
Have a consistent issue tracking system for Twitter4R. Previously it has appeared inconsistent to users of the Twitter4R library where to submit defect reports or new feature requests to. So I have had to field these requests from the Google Groups mailing list to personal emails and also @twitter4r replies on Twitter. I plan on using GitHub's new issue tracking system for this as it seems a logical extension.
I plan on documenting things a little more obviously on the official website from now one. It appears many users did not want to go to the RDoc unless specifically pointed to it, even though a large majority of the library is fairly well documented in the RDocs.

Please be patient while I fix these process/marketing/workflow issues of the project and in the mean time I will also be tackling OAuth support as well. Again thanks for your patience thus far and tweet me on twitter at either @SusanPotter or @twitter4r if you have suggestions or questions about the direction. If you have specific queries on using the actual API, please join the twitter4r-users Google Groups mailing list and search the archives for a solution first and then if you can't find a solution, please post a message giving the group members your environment details and the version of Twitter4R you are using. Thanks! Susan Twitter4R Developer

The Scala vs Erlang Debate, Part 2: The Geek Off

2009-04-20T15:47:00.028-05:00

In the first part of The Scala vs Erlang Debate article series, I discussed various ways to go about adopting either Scala or Erlang and in what scenarios it made sense to go with one over the other from the manager's perspective. In summary the major advantage of Scala is that it runs on the JVM so it can access pre-existing Java libraries your enterprise has already developed. This is also Scala's major disadvantage because it makes a rather large assumption (which in my opinion would be incorrect the majority of the time) that the Java libraries being used are thread-safe. Erlang on the other hand doesn't allow you to integrate in the same OS process with Java libraries, because Erlang was built from the ground up with scalability and fault-tolerence in mind and ideally you shouldn't put Java into the run-time stack because it compromises Erlang's fault-tolerance features significantly. Instead the preferred approach when writing the biggest bottlenecks of your system in Erlang and integrating with pre-existing Java libraries/system is to use SOA to create separate, but integrated systems that are easy to swap out and upgrade at a later point. In my view, you should be refactoring your enterprise architecture by defining services anyway, since this will decouple systems and remove a great deal of headaches in IT maintenance and deployment. Today I wanted to geek out and differentiate the two environments on a more technical level for the middle/technical managers.

Scala: Technical Overview

Scala is an objected oriented, statically typed language with many functional programming capabilities built-in. However, it must be noted (because few Scala documents do this) that Scala is hybrid actor programming language and therefore a number of assumptions cannot be made. So by being a hybrid functional language it losses some of the benefits. As well as it's Java run-time support on the JVM it is also supported in Microsoft's Common Language Runtime (CLR) so .NET integration, in theory, is a breeze. Again this also assumes that the .NET libraries you use from Scala are thread-safe. On a related topic, .NET also has F# available, which is Microsoft's attempt at a hybrid functional programming language also. I have no comment on F# myself, since I have zero experience with it. Please chime in on the comments section if you have experience with F# and are able to contrast it with Scala.

Erlang: Technical Overview

Erlang on the other hand is a dynamic, purer functional programming language that supports light-weight processes in its own run-time environment (separate from OS processes), which send messages to each other asynchronously. One major side effect of being purer in its actor treatment is that there is virtually not state. There is no way to pass "state" around (a couple of minor exceptions), but there is a lambda calculus optimization that can be taken advantage of that will ease away our troubles called tail-recursion. Tail-recursion only partially works in Scala. For many developers that only have experience in procedural and Object-Oriented (OO) programming languages like Java, Python, Ruby, C++, C, Perl, C# this sounds completely insane. Two years ago I may have agreed with you, but today life is much simpler without state in my Erlang travels than when I had to write thread-safe Java code and I wish I had discovered this concept earlier. I will not pretend this notion didn't take some getting used to but today I see things a lot clearer and I am thankful for that clarity.

Scala: Concurrency

Concurrency in Scala is modeled using a concept of actors, except there are two types of actors: thread-based and event-based actors. Since Scala was implemented on top of the JVM, which is stack-based, Scala had to compromise on Erlang's one actor approach. This is because native threads, which the JVM uses for concurrency support, are much heavier-weight than Erlang's lightweight processes thus reducing the number of thread-based actors that could be launched inside one JVM. So Scala created this concept of a very peared down event-based actor, which does not have all the functionality of thread-based actors but does not incur the penalty of needing to be backed by a native thread. To be as complete as I can be in this section I should also note that Scala treats everything as an object, even though it has these two types of actors. Actors in Scala are just objects. In turn this may create duplication between objects and actors.

Erlang: Concurrency

The Erlang VM was built from the bottom up with lightweight processes (a.k.a. actors), therefore it is stackless and much better at lightweight concurrency. Erlang also has a built-in mechanism for load balancing these lightweight processes across all available CPU cores. This means Erlang has a massive advantage over JVM based functional programming languages at getting the most out of multi-core hardware, which is at the crux of the problem domain that software needs to address going forward.

Scala vs. Erlang: Performance

In terms of linear performance (sequential) Scala has been shown to beat Erlang in certain cases. However, concurrent performance consistently shows Erlang beating Scala, sometimes by wide margins on multi-core hardware. Not surprising considering what we learned in the concurrency sections above. When considering the problem we are trying to overcome - optimizing software on multi-core hardware - it seems clear which of these benchmarks are more important! :)

Conclusion

I think it is clear, which side of this debate I firmly stand, but I have tried to demonstrate both good and bad traits of each programming language from a technical perspective that might translate into a business case or reason for choosing one over the other in your organization. If I have time I might do a part 3, which will talk a little about syntax and DSL support. This might round out the argument since it could be argued that Scala wins in the third part by a hair, but all is not lost for Erlang if the underlying problem you need to solve is related to optimizing software for modern hardware and building robust systems at the same time. My vote still goes to Erlang despite the lack of built-in features that make designing DSLs easier, but it is not difficult to build your own DSL engine in Erlang either.

Limitations of this series:

There are quite a few topics I left out of this post that might be of interest for some applications or services. Namely:

Hot swapping: Erlang has great hot swapping capabilities for production environments that minimizes or possibly eliminates downtime of systems. This feature in Scala is not close in capability or reliability at this point.
Garbage collection: without getting into too many details Erlang probably wins this section by a hair (not significant to write home about). Long sweeps will not be a problem in Erlang and only a problem in JVM languages on rare occasions.
Distributed nodes: The clear winner here would be Erlang because built-in to the environment are all the multi-node features that make distributed systems a breeze. Scala on the other hand has different frameworks and libraries to facilitate distributing nodes of logic on different hosts.
[Added Nov-2009] Frameworks: As far as I know, Scala doesn't have an OTP killer, not even a contender to compete with Erlang's Open Telephony Platform (OTP). Sure it will take some time, even Haskell, which has been around much longer than Scala is slightly jealous of OTP (I saw the way Don Stewart looked at me when I asked him if Haskell had framework similar to OTP;) ). Supervision trees along make it stand out as the best thing since sliced bread, at least for this particular software engineer.

Resources:

The Scala vs Erlang Debate, Part 1: The Manager's Overview

Corrections (added February 2010):

Erlang is not a pure functional language. However, it does not have a polluted (impure) treatment of actors like Scala:) Some think this is a good-enough approach to be able to leverage the JVM, when running on the JVM is really important, perhaps 4 times out of 5 it might be "good enough". I just wonder if it really matters for most projects that you run on the JVM!?
"Scala's lack of tail recursion is entirely due to a trade-off. The JVM doesn't do tail recursion itself. It is possible to add tail recursion on top of the JVM using various tricks, but they are expensive. SISC does its own stack based heap and Kawa uses trampolining." -- James Iry

The Scala vs Erlang Debate, Part 1: The Managers Overview

2009-04-15T09:37:00.017-05:00

Note: The discussion is primarily between Scala and Erlang, but other languages could potentially be substituted in the arguments for and against. Scala stands for any language that attempts to solve the multi-core issue software developers face with threads and Erlang is a language that has light weight processes which solves the biggest issue of going with a multi-process solution in software. The debate started with the realization that Moore's Law died a while ago. Tomorrow (as we see it now) is destined for multi-cores. A problem of massive proportions now faces software engineers, since hardware engineers changed the rules on us. Now I am not giving hardware engineers a hard time, I am simply pointing out that there has been a massive shift in the way hardware is built, which directly affects the performance and reliability of the software which runs on it. It is not a transparent change that software engineers can ignore. With that background laid out, as software engineers, we must appreciate the problem we now have to tackle, both server-side and client-side developers alike will need to tackle these issues promptly. However, this article only considers server-side software as there is where my specialized expertise lies. This part of the article is the high-level. I'll get into the specifics in the second part. So the question now is how do we leverage the advantage given to us by new hardware progress of multi-cores? There are two fundamentally different problems to solve: software products/projects that already have large complex systems already built in languages that support threads half-decently such as in Java and JVM languages including Scala; but there are also those startups that don't have baggage and can start with a better long-term solution by using a true concurrent language like Erlang.

The Legacy Enterprise

The problem with utilizing JVM threading (whether directly in Java or via other languages built on the JVM such as Scala) is that third party code you use or existing Java/JVM code written isn't necessarily thread-safe. You might have all your business entities and models defined in Java, but can it really be threaded? The pain of migrating or porting your existing system from a JVM language to another truly concurrent language like Erlang might be much greater than dealing with threading issues in just a few key bottleneck areas that will increase throughput and CPU utilization enough. In this case I doubt many pragmatic engineers or architects will advocate starting from scratch in a language you know little (if anything) about. However, identifying those key pain points might take some time, but will be a necessary cost if you are serious about solving this problem in the near-term using a solution like Scala. However, what these engineers and architects should stress to management is that this is likely to become a monkey patch. There will be a tipping point where moving to a pure concurrent language like Erlang would be more beneficial that sticking to the monkey patch and living on the JVM forever, no matter what Sun tries to tell you. There are hybrid solutions for the medium-term, that depending on the project/enterprise specifics may be a good solution until you are able to fully port the code to a langauge like Erlang (or whatever the trend may be at the time you get to it). SOA principles can be used to design and define a meaningful architecture that will declutter the system and allow you to tackle just the bottlenecks of the system in an Erlang port/rewrite and still give you (the manager) a low risk PoC project so you can evaluate Erlang in your enterprise. Having worked on a number of large enterprise projects where upgrading the "legacy" system was the goal, I know full well that management are likely to prefer the monkey patches (i.e. Scala where the pain of rewriting components in a PoC way doesn't exceed the pain of threading your existing code at the bottlenecks), especially when the middle/technical managers do not highlight the drawbacks of the "solution" adequately. Three to nine months later the management (especially middle management) are hurting big time, but it is not usually the most appropriate time to do your "I told you so dance". Trust me!:) The the monkey patch isn't an ideal solution when you are simply trading one complexity of equal of greater value for the other and in 2-5 years the likelihood that there will be many more skilled engineers and architects that truly understand and have solid production experience in a truly concurrent language like Erlang is high in my estimation, so making a move to such a language/environment would not be as risky at that time. However, for 2-5 years you have been getting little out of your hardware and software together unless you overbuy hardware. In fact, in the case where legacy code already exists, there is never really an ideal solution. However, starting out with meaningful PoCs and structuring low risk exit points after conducting objective evaluations of these projects is key to the success of every "legacy solution". If in doubt and your legacy system isn't showing signs of hurting yet, you might want to wait and beef up on hardware. This is a transition period and if you are in a conservative IT enterprise this might be a better near-term approach.

Startups

Now for the startups or those starting afresh, there is mixed news: very good news and not-so-bad news. Green field projects, in theory, are always the best. You can get everything right from day one....well at least you can hope to do so (let us pretend for this article). My advise for your team is to take the plunge to Erlang. It is actually relatively low risk to do so for the following reasons:

It is a mature language and environment already tested by large and complex real world systems at telcos where reliability and scalability were and are top priorities.
There are a few good books out there on that will help you convert to Erlang and Functional Programming (see resources section below), although documentation and education on Erlang, generally, is somewhat lacking.
A framework for services has been designed, developed and put into production called OTP, which stands for the Open Telephony Platform and actually nothing specific to Telephony applications in it, it was simply developed by a Telco.

Erlang's approach to concurrency is interesting and takes some getting used to as it is a pure functional programming language unlike Scala which is a hybrid. Now the bad news is that there are very few people that know Erlang (or other pure concurrent languages), you need to hire at least one specialist with Erlang experience to guide your team to avoid the potholes and perhaps even stop you before you get to the edge of a cliff. They might not be cheap, but starting out it will mitigate project risk significantly. Another negative on the Erlang side of the argument is that you will not be able to use your Java classes straight from within Erlang (in a meaningful way - I'll cover this in the next part). The positive about this negative though is that you can encapsulate your Java code in an SOA way so that you can decouple and declutter subsystems within your enterprise, which is an important task to do regardless of which legacy solution you choose. Finally, good luck and let me know how it goes!

Resources

Programming Erlang: Software for a Concurrent World by Joe Armstrong

This book, for the moment, is the definitive guide for Erlang. If you have already decided to go the Erlang route then you will not mind (or perhaps not even notice) the "selling" of Erlang throughout the book. Overall this is an essential book to anyone learning Erlang at this point.

Real World Haskell

While Haskell isn't either Scala or Erlang, if you do not come from a functional programming frame of mind you will need a good book to introduce you gently into the realm of functional programming. This book does a pretty good job. While the syntax for Haskell is different to both Erlang and Scala, the hardest part of moving to functional programming isn't the syntax. Trust me on this one:)

Also consider me, Susan Potter, a resource if you need an Erlang software engineer with high-level enterprise architecture background in Java/J2EE/JEE and Ruby/Rails production environments. I am developing a risk management product in Erlang right now and I am available as a consultant through my consulting firm Finsignia, which mostly works with financial services firms such as investment banks and hedge funds, which is where my business domain expertise lies, but also works on designing, building and deploying reliable and robust server-side solutions for any industry.

CUBarCamp: Champaign-Urbana's Tech Meetup Update

2009-04-04T20:13:00.005-05:00

This is just an update blog post to let geeks in and around the Champaign-Urbana area know that the CU BarCamp is now scheduled for April 25th, 2009. The meetup will start at 2pm CDT and will be located at the Collective Turf Coworking space:

110 W. Main Street Urbana, IL 61801

It is right between the Heartland Gallery and Priceless Books and opposite the Urbana Business Association office and Crane Alley.

Want to participate?

If you want to volunteer, present or demo any topics at this BarCamp please update the CUBarCamp wiki page and join the Google Group for the CU BarCamp event. Hope to see you there!

YUI MenuBar: A Facebook-like Application Bar

2009-03-27T18:59:00.005-05:00

Below is a snippet of HTML code with embedded Javascript code that will create a top anchored menu bar very similar to what you see at the bottom of the Facebook page:

<!-- Add the following to your HTML/HEAD -->
<link rel="stylesheet" type="text/css" href="http://yui.yahooapis.com/2.7.0/build/fonts/fonts-min.css">
<link rel="stylesheet" type="text/css" href="http://yui.yahooapis.com/2.7.0/build/menu/assets/skins/sam/menu.css">
<script type="text/javascript" src="http://yui.yahooapis.com/2.7.0/build/yahoo-dom-event/yahoo-dom-event.js"></script>
<script type="text/javascript" src="http://yui.yahooapis.com/2.7.0/build/container/container_core-min.js"></script>
<script type="text/javascript" src="http://yui.yahooapis.com/2.7.0/build/menu/menu-min.js"></script>
<script type="text/javascript">
YAHOO.util.Event.onDOMReady(function () {
  var oMenuBar = new YAHOO.widget.MenuBar("menubar");
  oMenuBar.render();
});
</script>

<!-- in your HTML/BODY modify the following for your own "Facebook Application Bar" menu -->
<div id="menubar" class="yuimenubar">
  <div class="bd">
    <ul class="first-of-type">
      <li class="yuimenubaritem first-of-type">
        <a class="yuimenubaritemlabel" href="#file">Applications</a>
        <div id="file" class="yuimenu">
          <div class="bd">
            <ul>
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#">Notes<em class="helptext">Ctrl + N</em></a>
              </li>
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#">Links<em class="helptext">Ctrl + L</em></a>
              </li>
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#open">Photos<em class="helptext">Ctrl + P</em></a>
                <div id="open" class="yuimenu">
                  <div class="bd">
                      <ul class="first-of-type">
                        <li class="yuimenuitem">
                            <a class="yuimenuitemlabel" href="#">Upload Album</a>
                        </li>
                        <li class="yuimenuitem">
                            <a class="yuimenuitemlabel" href="#">Browse Albums</a>
                        </li>
                      </ul>            
                  </div>
                </div>            
              </li>
            </ul>
          </div>
        </div>
      </li>
      <li class="yuimenubaritem first-of-type">
        <a class="yuimenubaritemlabel" href="#"edit>Something Else</a>
        <div id="pim" class="yuimenu">
          <div class="bd">
            <ul class="first-of-type">
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#"> Bla 1</a>
              </li>
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#"> Bla 2</a>
              </li>
            </ul>            
          </div>
        </div>
      </li>
    </ul>
  </div>
</div>

For further details about customizing the YUI MenuBar look and feel you should visit the API documentation

CUBarCamp: Champaign-Urbana's Tech Meetup 4/22 (tentatively)

2009-02-24T00:04:00.003-06:00

Just a quick post to mention that with the assistance of Brian Marick and hopefully a few more people in the weeks to come, BarCamp is coming to the Champaign-Urbana area and is to be held at the Collective Turf Coworking space in downtown Urbana (110 W. Main Street, Urbana, IL - opposite Crane Alley and above Priceless Books). To keep updated on the latest developments and participate in the event organizing join the CUBarCamp Google Group. Hope to see you there!

Coworking Space in Champaign Urbana (Illinois)

2009-02-09T15:26:00.006-06:00

Since I moved to Champaign-Urbana, IL (aka CU) in 2005 I have been looking for work space that would provide me with a more productive environment and social yet professional outlet to work in since I work primarily on telecommute contracts and lack face time with clients. Today I decided that I will setup a local coworking space for solo entrepreneurs and freelancers in and around the Champaign-Urbana area to fulfill my needs as a consultant and fulfill my new venture urge of late. Essentially coworking is defined as the following:

Coworking is a movement to create cafe-like community/collaboration spaces for developers, writers and independents.

This is exactly how I imagined my dream work space since I work a lot in coffee shops all around town throughout the work week and even during the weekend sometimes and find it surprisingly (initially) productive. I have seen these types of office spaces in larger metro areas and the terminology used tends to be different from place to place. Some terms used to mean a community of independent professionals sharing office space, facilities and amenities that I have come across on my travels include:

Coworking or Co-working (obviously)
Collaborative work environments (not used in the context of a company job description:))
Work Space Cooperative (this might be organized at the top levels slightly differently though)
Shared Work Space
Serviced Office Space

There are probably some slight variants on the definitions of these as compared to that of coworking, so some places will have more structure and a more formal setting, whereas coworking itself strives to create a more creative and collaborative environment for independent professionals. If you are in the CU (East Central Illinois) area and would be interested in working in such a coworking space, please drop me a line at any of the following locations:

twitter: @SusanPotter
email: me at susanpotter dot net
facebook: My Profile | Champaign-Urbana Coworking Group

Are you a freelancer or entrepreneur in another area or region that is part of a coworking environment experience there? What do you like about your coworking space and why? What would you change about it if you could?

GEdit Extensions Project

2009-01-14T23:54:00.006-06:00

Just a quick post from a long blogging hiatus to let people know that I launched gedit-extensions where I will maintain a set of commonly used GEdit snippets and eventually plugins and tools that others might find useful. You can view the snippet source at the GitHub gedit-extensions repository. I plan on adding documentation to the gedit-extensions wiki in the next couple of days, but for now see the sneak peak in the Ruby example snippets at the end of the post. The name in the comments directly above the example snippet is the shortcut you can use. Type the first few characters of the shortcut and then Ctl+Space key combination to pull up all shortcuts the characters match so far. Choose the relevant shortcut from the drop down list. If the snippet contains variables the first variable will be highlighted for your to fill in, then press TAB to navigate to the next variable to fill in. As you will see most of the shortcuts so far are Ruby, Rails, RSpec or Cucumber specific since there are a few Python snippets I need to untangle from our in-house code references.

# shebang
#!/usr/bin/env ruby
# absrequire
require(File.join(File.dirname(__FILE__), "config", "enviornment")

# habtm
has_and_belongs_to_many :idiots
# unique
:unique => true
# habtmthru
has_and_belongs_to_many :idiots, :through => :village

# belongsto
belongs_to :village

# createtable
create_table :sessions do |t|
  t.column(:name, :string, :limit => 34)
  t.column(:description, :string, :limit => 255)
end

# addindex
add_index(:sessions, :name)
# changecolumn
change_column(:sessions, :name, :limit => 70)

# removecolumn
remove_column(:sessions, :name)
# irreversible
raise ActiveRecord::IrreversibleMigration, "the reason the migration is irreversible"

# resource
class PersonResource < ActiveResource::Base
  self.site = "http://domain.com:8080/"
  self.element_name = "contacts"
  self.prefix = "v1_0"
  self.user = "supo"
  self.password = "geditrocks"
  self.timeout = 60
end

# describe
describe PersonResource do
  before(:each) do
    @resource = mock(PersonResource)
  end

  it "should bla" do
    
  end

  after(:each) do
    @resource = nil
  end
end

# specmodelerror
it "should have an error on name" do
  @user.should have(1).errors_on(:name)
end

# specmodelrecords
it "should have 50 records" do
  @states.should have(50).records
end

# spectemplaterendered
it "should render 'common/user' on get '/user'" do
  get('/user')
  response.should(render_template("common/user"))
end

# specresponsetext
it "should return text 'You were signed out.' in response to post on '/signout'" do
  post('/signout')
  response.should have_text("You were signed out.")
end

# specresponseredirect
it "should redirect to action 'index' on post '/users'" do
  post('/users')
  response.should redirect_to(:action => 'index')
end

# specroutefor
it "should map #index to '/users'" do
  route_for(:controller => "users", :action => "index").should == "/users"
end

# specparamsfrom
it "should generate params for #update" do
  params_from(:put, "/users/1").should == {:controller => "users", :action => "update", :id => "1"}
end

# spectagexists
response.should have_tag('ul.menu#sidebar')

# specnestedtagexists
response.should have_tag('ul.menu') do
  with_tag('li', 'my menu item')
  # specchildtagexists
  with_tag('li', 'my second menu item')
end

# specblocktagexists
response[:sidebar].should have_tag('ul#sidebar')

# mockviewhelper
template.should_receive(:curent_user).and_return(mock(User))

# specsuccess
response.should be_success

# specredirect
response.should be_redirect

# feature
Feature: Search Jobs
  In order to find relevant jobs
  As a Job Seeker
  I need to find jobs relevant to me

  Scenario: Location Search
    Given a search form
    When I enter my zip code
    Then I should see all jobs in and around my zip code
  # scenario
  Scenario: Skills Search
    Given a search form
    When I enter skill keywords
    Then I should see all job descriptions that mention those keywords

Free Coffee for Voting

2008-11-03T10:50:00.008-06:00

As if anyone needs any more enticement to vote this election season, Starbucks has decided to reward people who cast their vote Tuesday November 4th, 2008 with a free Tall coffee. I do not believe you have to vote for anyone in particular:) Just go and vote. While you are enjoying your free coffee at Starbucks, perhaps explore the latest Merb 1.0 release candidate. To install simply do the following:

$ sudo gem update --system
$ gem --version # make sure it is at least 1.2.0
$ sudo gem install merb

Now to create a new Merb application do the following:

$ merb-gen app your-app-name-here
$ cd your-app-name-here
# now start editing the files you need to edit for you app.
# I promise more tutorials will be coming once I have 
# free time

Book Shelf Clearing

2008-09-27T21:52:00.020-05:00

In an effort to reorganize my office, I am clearing off space on my bookshelves and selling books (on Amazon marketplace) that I have already read. Since they are all software development related I thought I would list the most relevant below in case any blog readers are looking for books on the subject for cheap:

[SOLD] ~~Deploying Rails Applications: A Step-by-Step Guide (Facets of Ruby)~~ (buy it new or used)
Very good for the beginner or intermediate Capistrano recipe writer. It also contains some good pointers at the end of the book on benchmarking. Released relatively recently (within 3 months I think?)
SOA in Practice: The Art of Distributed System Design (Theory in Practice) (Published Aug 2007)
This book is more for the unRESTful SOA types than the RESTful folk like me. The author obviously has some old-time experience, so if you are into old school SOA standards (WSDL, SOAP, UDDI) then this would provide a lot of good architect-level areas to consider, which are often ignored including: service lifecycle, versioning, security, service management and more. One gripe I had with the book is in the versioning section. Let's just say his preference for naming is definitely NOT my style:) Otherwise this contains good reading material for the architect just getting into SOA in an enterprise setting.
Rails Cookbook (Cookbooks (O'Reilly))
Great for starting Rails projects with limited Ruby on Rails experience. It covers Rails 1.2, but most recipes will only require minor changes.
The Art of Agile Development
Covers all aspects of agile development from planning to delivery. If you aren't already an agile development ninja, this is in my top three agile books.
Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL
Last year I had to work (for a short while) with PHP in the crawling/bot arena. This book contained got me developing bots and crawlers in no time in PHP.
Ajax on Rails
This would be a decent book for Rails developers that aren't believers in Unobtrusive Javascript (UJS). I have to purge this book from my bookcase now that I am a big UJS convert:) I hope you understand. I got this when it first came out, before I had seen the light. Seriously though, if you don't care about UJS principles, this book covers good ground on using AJAX Rails helpers. You should realize that the book was written pre-2.0 (Rails release that is).

Also those wanting to branch out on their own as a consultant, the following books might be for you as I found both very helpful when first starting my software consulting practice in 2001:

Getting Started in Consulting
If I could only recommend one of these two consulting books it would be this one hands down, since this is more about general consulting principles and practices and the second book might be slightly dated in the initial chapters, because both were written before the software outsourcing/offshoring mania.
Getting Started in Computer Consulting

PS I have been using Amazon Prime for two months now and love it. You can try Amazon Prime FREE for one month if you like too.

Tips to Enterpreneurs Hiring Ruby on Rails Consulting Firms

2008-09-22T10:20:00.041-05:00

I know I am taking a break from my DataMapper/Merb series of posts today, but I have seen enough fakers out there in the Ruby on Rails world that I just have to get a few things off my chest: Below is a list of things to avoid when hiring a Ruby on Rails consulting firm:

Overexcited marketing speak. Drop any firm that has something like the following on their website: "We believe Ruby on Rails is the most elegant, powerful programming language on the planet". These people might know a thing or two about configuring WordPress or Drupal or perhaps only Photoshop, but definitely no serious Object Oriented development of large scale Ruby server applications. Basically anyone that calls "Ruby on Rails" as a whole a programming language has an inadequate skill set to build a non-trivial web application. In addition anyone that knows more than three non-similar programming languages will know Ruby is not "the most powerful programming language on the planet". Sorry I had to find a quote on the web to prove I wasn't exaggerating. There are many more fakers on the web too, so look out! Remember you aren't looking for people using the most powerful programming language, you want results.
Agile this, agile that, agile everywhere. Beware of firms using the word agile like there is no tomorrow. Some use is fine, but ask them what they mean by agile. And what benefits you, the client, will receive from an agile team. Don't get me wrong if a web application is built in an iterative fashion with lots of client involvement and sign off, with lots of useful automated tests/specs that are run often then this is almost certainly a great thing for the project. However, beware of the firms or freelancers that are using agile simply as a buzzword. (Note: some firms like mine use agile a lot in web page titles for SEO purposes, so not all firms that use agile a lot in titles are faking. So make sure to ask firms what they mean by the word agile when they use it).
Why Rails? If any firm tells you that a web application written using Django, TurboGears, Merb or other web frameworks will never be as good as a Rails web application, then drop them off your consideration list immediately. The same is true if the converse is said. If they answer with something more measured and sensible like "our engineers have found developing applications in Rails to be more productive than X, Y and Z, which we previously used, and it provides us with the right amount of control/flexibility without us needing to worry about all configuration than previous frameworks" then at least they aren't misleading you just to get a sale. Of course, you might be looking for other points in the answer (other attributes of a web framework that would make business sense like active community, skillset, etc.). Also in general you should look for measured answers to all your questions, not answers that lack insight and promise they can do EVERYTHING under the sun for you in 2 weeks time. You will be disappointed if you hire that firm or freelancer.
Expert in everything, expert in nothing. Consulting firms and/or freelancers need to choose a specific set of technical competencies so they can be experts in those areas. Beware of consulting firms or freelancers that claim to be experts in every framework, programming language and operating system.
Ubiquitous buzzwords. Beware of too many buzzwords. Even if you know what a buzzword means, ask the person using too much jargon to clarify so you can evaluate whether they really know what they are saying.
Everything AJAX. I previously worked through a consulting firm on the west coast (I'll not name names). The consulting manager who led project management functions on all the firms' projects was convinced that if we just put an AJAX form in here and there it would solve all our problems. My experience with this firm wasn't overly positive. I like the people on a personal level, but their work quality and focus tended to create hard to use web applications. Beware of people who think AJAX is the answer to everything. It has its place, but don't go overboard. PS Make sure the people you hire write unobtrusive Javascript rather than embedded Rails Prototype helpers. UJS has many benefits including graceful degradation (benefiting those that do not have Javascript enabled), avoiding browser incompatibilities and improved code organization (separation of concerns). Users of your web application will appreciate the forethought and your developers can be more productive!:) Win-win.
Novice.has_many :problems Look for previous Object Oriented modeling experience for the engineers on the project team. Even though ActiveRecord (the default Ruby on Rails Object Relational Mapping framework) provides a higher level of abstraction for ORM associations than other frameworks, don't be fooled into thinking anyone can model your business domain correctly without experience and/or know-how. Many developers who are used to using Hibernate or TOPLink (as only two examples) before in Java or Smalltalk respectively will have likely done a lot of OO modeling. Ask them about the most complex business domain they have modeled previously to be able to guage their skill level and experience in this area. This point is most important in applications that have more complex business domains.
Why should I care about your data? Look at my cool Photoshop spinning logo. Rails consulting firms and/or freelancers should have a good understanding of the importance of data modeling. Even if they do not have specific data modeling expertise they should at least know that indexing appropriate columns and defining column size limits where appropriate can save not just storage space, but also shave time off queries (depending on the query and usage scenario). That way when your application gets to the point where the database needs to be fine tuned, the data architect you hire will have an easier job than if your schema was designed by a PHP weekender that thinks the Drupal or WordPress database schema is finely tuned (sorry, it's horrendous and yes you should know that anyway). This is a very important consideration if your application is supposed to hold a considerable amount of data and/or if you have strong reporting requirements. As a side note, make sure to ask them if different database schemas are sometimes needed for querying large data sets (classical reporting requirements) versus CRUD usages. The answer should be yes and you should ask them to explain this a little to see how much they really understand.
I can design, develop, deploy and manage perfectly. Usually firms and freelancers are much better in one or two areas than all equally. Some are better at graphic design and got into development to earn $20/hr extra. That's fine, but you should know what you are getting yourself into. Ask what their strengths and weaknesses are. For example, I am excellent at developing and very good at deploying and systems engineering. I can use Photoshop and Illustrator (just like graphic designers can copy Rails code from forums and paste it into their editor), but ask me to create a visually stunning logo that encapsulates your company's image and I am lost (just like the graphic designers that can't write original code themselves). Know what is most important for your application needs and hire the appropriate people. Usually outsourcing graphic design to a design firm and hiring a flexible developer that can work with external teams well, gets you the best value for money. Throw away the firms and people that don't have any weaknesses. They are lying.

One point I would like to make to round this piece off is that if you find a young, eager engineer with higher than average aptitude, he/she can really do a lot for your project. However, you should expect them to make a few mistakes. Nobody is perfect and without prior experience with large scale server applications it is virtually impossible for someone not to make mistakes. If you can afford a few mistakes here and there, then this might be the way to go. Give them an equity stake if capital is an issue for you, they are more likely to take a risk with you than a seasoned consultant with a mortgage to pay or spouse and children to feed. However, never hire someone who claims to be a Ruby/Rails expert at a high rate if they simply copy and paste sample code from Ruby/Rails forums online (you would be surprised how many there are out there). It's a terrible waste of your capital and frankly it takes work away from better firms and/or freelancers out there and you will end up with badly written software that runs poorly. As a final note, hire a firm, freelancer or in-house engineer that has similar energy level and compatible personality to yours. Or at a minimum hire a firm or freelancer that you can trust and has been recommended, but not just through popularity contests like those at WorkingWithRails.com (I know three freelancers who are excellent Ruby engineers with only have 1 recommendation on the site and I have worked with one person that has more than ten recommendations and consider them subpar to most Ruby/Rails developers I know).

DataMapper: Flexibility of mapping model attributes to table columns

2008-09-04T10:13:00.015-05:00

I know an esteemed ex-colleague of mine (even if he is an Apple zombie now - yes, becoming a fan of Apple on Facebook deserves some heckling;)) has reservations about the apparent unDRY-ness of DataMapper with migrations. I used to share his reservations before I wrote two applications using DataMapper. Neither Merb application was particularly spectacular, but both had to work with legacy database schemas. Oh the joys of crazy DBA naming conventions! Let's look at a brief example of DBA naming madness (some names were changed, but the conventions used are the same):

ordersTable:
  - uid_int: int (PK)
  - acc_id_int: int
  - amt_int: int
  - ref_code_string: varchar(64) natural key
  - desc_string: varchar(128)
  - entered_dt: datetime
  - changed_dt: datetime

In another table managed by a different group we had:

Account:
  - ID: int (PK)
  - Balance: int
  - UserID: int (FK)
  - EnteredDateTime: datetime
  - ChangedDateTime: datetime

I kid you not. Two different conventions, both relatively unreadable as Ruby attributes as they both violate popular coding conventions in some way. In ActiveRecord you would be left doing non-trivial coding to hide the ugly column names in the model and map Ruby-friendly names to the appropriate column. In DataMapper this is not so. Lending from the design pattern's primary purpose - to explicitly map model attributes to database columns to decouple the two - we can write something like the following using DM:


class Order
  include DataMapper::Resource
  set_table_name("ordersTable") #ridiculous name really, but what can you do if other older applications rely on this already?

  has(1, :account, :child_key => 'acc_id_int', :repository => repository(:accounts))
  property(:id, Integer, :serial => true, :field => 'uid')
  property(:amount, Integer, :field => 'amt_int')
  property(:reference, String, :field => 'ref_code_string', :key => true)
  property(:description, String, :field => 'desc_string')
  property(:created_at, DateTime, :field => 'entered_dt')
  property(:changed_dt, DateTime, :field => 'changed_dt')
end

class Account
  include DataMapper::Resource
  set_table_name('Account')

  property(:id, Integer, :serial => true, :field => 'ID')
  property(:balance, Integer, :field => 'Balance')
  property(:created_at, DateTime, :field => 'EnteredDateTime')
  property(:updated_at, DateTime, :field => 'ChangedDateTime')
end

This way we can write readable Ruby code with the model. Next time we can look at lazy loading: how to switch it on and off and when to use it for greater effect.

DataMapper does have migrations

2008-09-04T09:27:00.007-05:00

I wanted to bust a myth out there that DataMapper does not have regular migrations, just the auto migrations. At least in post 0.9 versions DataMapper has both. In other blogs about the topic I found quotes like: "DataMapper migrations pull your database structure directly from the Ruby code for your models, so there’s no need to write separate migration files...". This, in my opinion, gives the wrong impression to those that are worried about non-trivial schema migrations. The author of that quote appears to have never moved one column into multiple or vice versa (a non-trivial schema change), among other schema changes where a DataMapper auto migration would not work in a production environment. In merb you can just create a new DM migration with the following: $ merb-gen migration name_of_your_migration (Assuming you have set the use_orm setting to :datamapper in config/init.rb.) A simple DataMapper migration might look something like:


migration 1, :create_orders  do
  up do
    create_table(:orders) do
      column(:id, Integer, :serial => true)
      column(:amount, Integer)
      column(:completed, Boolean)
      column(:description, String, :size => 255)
      column(:created_at, DateTime)
      column(:updated_at, DateTime)
    end
  end

  down do
    drop_table(:orders)
  end
end

If you have problems with Boolean not being recognized, throw in the line: include DataMapper::Types to the top of the migration (yes, this needs some finessing obviously). Hope that helps a few people that may have been mislead by some blog posts inadvertently. Migrations are needed for many cases I come across where auto migrations just will not cut it without losing data in production (not a good thing if data matters in your application:)). BTW A minor annoyance of mine in Merb is that for DM migrations you need to use this ugly Rake task: dm:db:migrate to migrate to latest version. To rollback migrations you need to use this ugly thing: dm:db:migrate:down[version]. Where those using AR in Merb just keep using db:migrate and db:rollback. This is an area that needs cleaning up to promote ORM agnosticism the way Merb is supposed to do.

Next Generation Syndication and Publishing

2008-08-30T08:34:00.012-05:00

For the last year or so, I have been focusing on implementing RSS, Atom publishing and related standards for a particular client in the podcasting industry. Both RSS and (to a lesser extent due to its standard being finalized much later) Atom are being heavily used not just for blogs, podcasts, videos and multimedia content, but also used heavily for web 2.0 (or social media) activity "watching". Most people should be familiar with FriendFeed by now (if not, where have you been? really!). This month (August 2008) FriendFeed came out with their Simple Update Protocol (SUP) which sits on top of RSS/Atom publishing to make it easier for content consumers (very self-serving for FriendFeed, but at least they are coming up with some kind of "solution" instead of just bitching about the status quo) to identify feeds that have changed before they make a timed request for the feed every X minutes regardless of whether the feed has changes or not. In a nutshell, SUP requires a meta JSON feed that notifies consumers of newly updated feeds to publish data. The first time an RSS/Atom feed is subscribed to, the consuming agent (e.g. FriendFeed) will strip out the SUP-ID and the SUP feed URL from the xpath:channel/link element and associate this information with the relevant user/account or whatever. Then using the identifier for a feed returned by the meta JSON feed, the consumer knows the URL associated with that ID for feeds they need to be updated by. While this might solve some problems, it really does feel very much like a monkey patch. On the flip side I do understand to some extent why they took this path of monkey patching as opposed to coming up with something more revolutionary. Revolutionaries tend to get killed and only the second wave settlers reap the benefits of their blood, sweat and tears. However, I did want to mention SixApart's efforts to broadcast frequently updating consumable data in what should be a much more obvious way for former enterprise MOM developers like myself. The Six Apart Update Stream broadcasts over HTTP new public updates/posts as they happen. It's an interesting idea and much closer to the XMPP ideal that companies like Twitter have attempted. Any other former enterprise architects merely see this new trend in web [defacto] standards in publishing and consuming data as just another message oriented middleware (MOM)? And I am hopeful that finally the web has found a MOM in XMPP that can stand the test of public scrutiny.

NEVER use or trust Paypal

2008-08-21T17:20:00.005-05:00

For those that have been screwed by Paypal and need to find another venue to conduct business here are some alternative services:

For *some* of the many issues surrounding Paypal/Ebay policies and rules you can check out the following articles:

EBay's PayPal funds freeze plan draws fire (CNN Money)
Paypal Debit Card Disputes
Ripoff Report: Paypal

I have recently been royally screwed by Paypal. However, I have been advised not to comment on specifics until I play all my cards:)

LA Times Travel uses Twitter4R

2008-07-22T18:51:00.005-05:00

LA Times Travel is now using Twitter4R. You can see they sent messages via the Twitter4R for a few hours today (2008-07-22) on their Twitter timeline before changing the Twitter4R configuration to not sure which library/application they are using (or just waiting to get Alex to approve their application source code). Please let me know if you see Twitter4R sightings being used by other well known or cutting edge applications.

Serendipity: An original Pradipta's Rolodex member

2008-07-18T08:44:00.005-05:00

Last night I got the Pradipta's Rolodex message. Rock on Pradipta!:)

Twitter4R Development Releases

2008-06-30T22:04:00.003-05:00

Over the last week or so I have been putting Twitter4R sources in GitHub and getting the on-the-fly GemSpec to work "safely" enough for GitHub servers. Tonight (June 30) I decided I would announce Twitter4R development micro-releases available only via the GitHub gem server. To setup your environment to install the latest development release of Twitter4R, please read HowTo: Install GitHub Development Releases Official documentation will still exist through the official Rubyforge project site of Twitter4R, but any documentation related to current development tasks, features, bugfixes will now be documented on the Official Twitter4R GitHub Wiki. If you want to keep an eye on the Git repository, feel free to watch the mbbx6spp/twitter4r GitHub repository.

Rails rocks, X sucks: How Provincial

2008-05-20T14:12:00.015-05:00

I seem to be coming across more and more [Steve] Jobs-clones or simply no-thinking Apple fan boys lately. I don't mind the Apple fans that thought themselves about why they love their stylish new Macbook Pro, I have just had enough of the no-thinking variety of Apple zealots. This part of the Rails community appears to be a large population unfortunately. Because these fan boys are actually incapable of thinking on their own, they adopt other people's arguments and say things like: "Rails rules, X sucks because so-and-so said this...". Hmmmm. Need I say more? On the other side I recently came across a Django fan boy who tried to convince his non-technical manager that Rails "cannot scale period". Hmmmm. How original. A blanket statement like this is not only misleading, it is actually in many ways technically incorrect. This is why in middle school we learned how to qualify our arguments. Why is it that supposed college graduates cannot do this in their 20s and 30s? It is one thing to strip out technical jargon for non-technical managers to understand a situation, but it is quite another to over simplify and mislead them because you have your own tech-religion agenda. Be passionate, but admit to yourself and others when passion is getting the better of you. Why is acknowledging your chosen solution's weakness bad? Doesn't it actually make the solution implementation stronger if you have thought about scenario-based weaknesses? How about we each try to *qualify* our positions on technology solutions out there, rather than figuratively urinate all over others without any basic respect for them. I encourage criticism, but criticize constructively. Say why. On the flip side, we should also accept constructive criticism too. The first step to constructive criticism, need not necessarily be candy coating your critique, but having the right intentions behind your critique. For example, if your motivation for giving criticism is to start a flame war (sorry that is probably a very out of date term now), then you will probably get what you want. However, if you really want to help a project get better because you care, then that is the first step. In addition, the way you word and tone statements (as Twitter's Alex Payne should have learned all too well by now) has a big impact. For example, Alex Payne's statements about Rails and Ruby cost me a contract at a former Rails shop that had a conniving PHP zealot waiting in the wings to jump on anything and take issue with Rails to upper management. Lucky for him, it worked and he had 3 to 4 months of leading a team to miserable failure using Drupal as a "platform". Not necessarily a reflection of PHP (or Drupal for CMS applications since it was far from a CMS application), but a reflection of this calculating pseudo-coder's inability to lead a project. A "technical" manager who thought the kernel version on his Fiesty Ubuntu laptop was 7.0.4. Enough said!:) Overall, I hope we can start to have intelligent conversations about software stacks rather than "yours sucks ass". Because the latter in the end favors nobody. PS Merb rulez and Rails suckz!:) If you want to know why read my teaser entry titled Why Merb is delicious, though I plan on writing more soon. PPS Merb wouldn't be where it is today without Rails.

Why Merb is delicious

2008-05-06T08:49:00.010-05:00

Below are the reasons why Merb is more delicious than other MVC web frameworks I have worked with (including Rails):

On a diet...It is lighter weight than most, but still has enough oomph to implement common features with minimal code
Religion-free...It is ORM, JavaScript/AJAX and testing/speccing library agnostic.
The sum of it's parts...OK that was a lame segway, but parts is a mightily useful feature where a plugin is just total overkill. Just because Java calls something a "component" does not make it the general definition of a component in concept. And now look at the Rails plugin debacle today.
Precious gems...Plugins are created, distributed and installed in the form of beautiful RubyGems as opposed to the Rails plugin catastrophe with Gigabytes of duplicate disk space and lame version/revision control
It's exceptional...Merb actually thought about how to do exception handling in controllers before Rails came up with something and also came up with IMHO a better solution.
Speedy Gonzalez...Compared to especially Rails, Merb performs very nicely indeed.
Loose threads...Merb is thread-safe unlike Rails, so that means one process can handle multiple concurrent file uploads, where Rails cannot right now.

All together I think most people will find Merb to be a much better thought out (conceptually) MVC framework than Rails. Over the next few weeks I'll be talking a lot about Merb and it's delicious APIs, Plugins, etc. as it nears it's 1.0 release.

Comedy of Errors

2008-04-04T10:23:00.001-05:00

Does anyone else find it comical that the Chicago Java Users Group website is written in Rails? And apparently not very well if it is not only sending a HTTP 500 error, but still using the default Rails generated 500.html page. Poor all around. An IM buddy mentioned it might be JRuby, but I do think it is still very ironic considering the hostility from the Java world regarding Rails in general (whether Ruby on Rails or JRuby on Rails). I just had to share!:) Update: Someone fixed it shortly after I reported it to CJUG.
Update: Actually it was only temporarily fixed, it is back to the default 500 Rails error page.
Update: In addition they have exposed their SVN information. See the public entries and from that information you can find their non-password protected source code and check it out if you like. If anyone from CJUG is reading, you can fix this by reading my blog entry from two months back called Preventing Information Leaks, Part 3. Although it might be useful to read part 1 and part 2 as well. Cheers! PS I also sent an email to info at cjug.org and received an email delivery error.
Update: I may have read too much into the SVN repository information exposure in this specific case since this is an open source Rails application project (I only just realized), thus the non-password protected repository, but I still stand by the rest!:) It is also not good practice to expose such information from the outset of deploying an web application.
Update (1.5 hours after trying to notify JUG and posting the blog post): The site has been fixed now for over 5 minutes, so good job fixing it CJUG!:)
Update (8 minutes later): Ooooops! It is back to the default Rails 500 error page. Perhaps it is 2 out of three bad FastCGI processes that need to be killed on the server? HTH someone at CJUG, but since I can't look at the system I can't say for sure.

Coding with Explicit Intentions

2008-04-01T10:34:00.008-05:00

Recently I have had the [mis]fortune to be working with PHP for integration purposes for a client. A partner of my client's (in the advertising space) sent us some PHP code to add to our servers for resolving the content type of a file based on its extension. The line of code in the PHP script they sent over that was supposed to determine the file extension was:

  $ext = substr($file,-4);

As you can see they are making a pretty big assumption - that the extension is exactly three letters long. The media files that this script *may* support in the future are: Real Media (.rm), MPEG (.mpeg), Quicktime (.qt), etc. These extensions are not exactly three letters long. Why would someone want to write code that is quite likely to fail and also not communicate more explicitly their intentions? This partner already doesn't support PHP < 4.0.3, so why not substitute the line above with:

  $ext = ".".pathinfo($file, PATHINFO_EXTENSION);

It is standard, will never fail (unless there is a defect in the PHP version you are using for the standard function pathinfo, but what can you do about that?) and communicates the explicit intent of the original author, thus improving code maintainability. Anyway, just a pet peeve. This is not by any means the only example I see on a day in day out basis, this just happened to be a great example for me to demonstrate my point. While this example shows just a minor one-line example, if developers introduce even only 2-3 of these un-explicit intentioned lines of code a day that do not always do what you might want it to do, then the codebase quickly becomes a mine field.

Waking Up From Hibernation

2008-03-30T21:49:00.003-05:00

Just a quick post to say that I will be waking up from OSS and blog hibernation at the beginning of May. I have a stack of bug fixes to apply to Twitter4R for a pretty large 0.3.1 release (in terms of bug fixes and supporting Rails 2.x compatibility). The next two major features on my Twitter4R v0.4.0 release radar are: (a) supporting Ruby 1.9 and (b) adding the newish "track" feature in some capacity (probably via XMPP instead of the REST API on Twitter.com). I also have a couple of minor features to add to the metafusion-crypto gem and finally release the first version of metafusion-rails. Then I will probably abandon OSS work explicit to Rails and write numerous Merb plugins/extensions. The catch up work will likely take 2-3 weeks into May and then hopefully my new venture workload will be significantly smaller to sustain OSS development over the summer months. Thanks for all your patience!

Preventing Information Leaks, Part 3

2008-02-12T10:44:00.000-06:00

In the third part of the how to prevent information leaks blog post series, we will look for unguarded "hidden" files where we can garnish quite a bit of information. Make sure you check out part 1 and part 2 on how to prevent information leaks in web applications before continuing.

Seek Unguarded Hidden Files

For web applications that use deployment tools that pull code from code repositories, you can find out quite a bit of information about the code repository (e.g. host, path, usernames, file listings), by finding unguarded hidden files. One obvious example especially in the SVN saturated Rails world is the path /.svn/entries and similar URLs. You will want to make sure the following URIs are not accessible on your public sites if you use Subversion (SVN):

/.svn/entries
/javascripts/.svn/entries
/stylesheets/.svn/entries
/images/.svn/entries

If you are using a superior SCM for your project like Git you ought to be looking for the following:

/.git/config
/javascripts/.git/config
/stylesheets/.git/config
/images/.git/config

If your project is still in the stone-age using CVS, then check the following:

/CVS/Entries
/javascripts/CVS/Entries
/stylesheets/CVS/Entries
/images/CVS/Entries

Adapt these patterns for your application framework default structures. The above is for Rails or Rails-based frameworks (e.g. Merb). When I first tried this last week on the Rails top 10 sites, three out of ten of these sites exposed their SVN information (and last week I had only checked SVN, no Git or CVS URIs). Since last week at least one of those sites have fixed the problem! What information is exposed in these files? Quite a lot of information. Do you want potential hackers knowing:

usernames of committers
code repository listings of files in a directory

All these things are not things you really want the potential hacker knowing. On my travels investigating the top 100 Rails sites I found one top 20 site that exposed enough information from their SVN entries file that I found a prototyping directory that included a "specs.rtf" document, which I was able to download. The document was not very uninteresting (at least not to me), but if they had written more in-depth specifications in the document, it may have served as a nice guide for a potential hacker to garnish enough information about their setup to take over! Moral of this story? Guard hidden files. I recommend "forbidding" access to all troublesome path regex patterns. For example, one of the LigHTTPd servers I administer has the following at the top of the configuration file:


$HTTP["url"] =~ ".*/\..*" {
  url.access-deny = ("")
}

For Apache httpd servers I use something like the following in the httpd.conf:


<DirectoryMatch "^.*\..*">
  ErrorDocument 403 /403.html
  Order allow,deny
  Deny from all
  Satisfy All
</DirectoryMatch>

Nginx configuration might look something like:


location ^~ /\..* {
  deny    all;
}

This guards against URLs like: /.svn/entries, .git/config as well as .htpasswd, etc. This means that only some script-kiddie hackers will mistakenly think you are using one SCM system instead of another, but they will never be able to know for sure which SCM you use or any information about your code repository. Forbidding access to all paths containing "/." somewhere in the URL is generally a good idea (IMHO) as sometimes people change web servers and leave "secret" files in the directory structure, but the new server doesn't know that, for example, .htaccess or .htpasswd is "special", so it will just serve it without thinking.

Preventing Information Leaks, Part 2

2008-02-11T13:04:00.000-06:00

Continuing the series that looks at how to prevent information leakage, today we look at the information leakage from web server HTTP headers.

HTTP Header Leakage

Let's look at the information potential hackers can get from HTTP headers from just a GET / HTTP request. Let us look at http://www.cnn.com:

$ curl -I http://www.cnn.com
HTTP/1.1 200 OK
Date: Thu, 07 Feb 2008 15:22:32 GMT
Server: Apache
Accept-Ranges: bytes
Cache-Control: max-age=60, private
Expires: Thu, 07 Feb 2008 15:23:23 GMT
Vary: Accept-Encoding,User-Agent
Content-Type: text/html
X-Pad: avoid browser bug
Content-Length: 90458

This example is pretty decent. What you want to look for is the Server HTTP header value. In this case it is just "Apache". Now it does identify the web server used, but it doesn't pinpoint the version being used. Now I am going to try a popular Rails website:


HTTP/1.1 200 OK
Date: Thu, 07 Feb 2008 15:25:34 GMT
Server: Apache/2.2.2 (FreeBSD) mod_ssl/2.2.2 OpenSSL/0.9.8b DAV/2 PHP/5.1.4 SVN/1.3.2 mod_vd/2.0 mod_fastcgi/2.4.2 proxy_html/2.5
Last-Modified: Thu, 07 Feb 2008 14:58:58 GMT
ETag: "4da437-36ac-b69b1080"
Accept-Ranges: bytes
Content-Length: 13996
Vary: Accept-Encoding
Content-Type: text/html

In this case we can see what OS Apache is running on and the version of Apache. Not only this, but we see all the enabled modules in Apache and their respective versions. IMHO this is too much information especially considering this site supposedly (at least as far as I know) host the site on a fully controlled environment (either dedicated or VPS with root access). Applications on shared hosts cannot help this much without assistance from the shared hosting company. I have a couple of very small sites that get little traffic on a shared host, so I appreciate this obstacle. Moral of this story: If you have full control over your environment you should always either change the "Server" HTTP header to something generic (e.g. "Apache" as in the CNN example) or disable it from being returned to the client. This setting is very easy in Apache, LigHTTPd and NGinx. I assume this wouldn't be difficult in LiteSpeed either, but I do not have configuration experience with LiteSpeed. Apache configuration:

Header unset Server

LigHTTPd configuration:

server.tag = ""

NGiNX configuration:

server_tokens off;

Also make sure there aren't any other headers that give away too much information. Especially look at the X- HTTP headers.

DataMapper vs. ActiveRecord

2008-02-09T12:17:00.000-06:00

Today I came across a thread on ruby-talk where I responded to the age old question of which Ruby ORM (specifically DM vs. AR). I responded based on my production use of AR, development use of DM and my understanding of Fowler's PoEA of the same names that I have applied in Java and Python countless times. As some readers of this blog might find this discussion interesting, I am linking to my response to the DataMapper vs. ActiveRecord discussion here. The executive summary (for the infinitely lazy) is:

It appears that if DataMapper (the Ruby library) is able to sufficiently hide enough database logic in more complex business logic scenarios, then the DM library might be more beneficial to use when using a legacy database schema where you are not able to create primarily isomorphic relationships between class attributes and table columns. Whereas AR would be a slam dunk and simpler to use in isomorphic schema scenarios where you have control over the database schema.

I will be returning to the Preventing Information Leaks series on Monday/Tuesday (it is already written, I just need to publish).

Preventing Information Leaks, Part 1

2008-02-07T10:04:00.001-06:00

Before I start I want to mention that the techniques suggested in this blog post are for readers to use to secure their own web applications. This does not mean that you can take over the server with these techniques without further hacking, but these steps provide a lot of information about the environment the web application is deployed to or developed using, unless system administrators/engineers or software engineers prevent this information leakage. Some of these techniques exploit settings that are usually found in Rails when default settings are used, but web applications written in different frameworks may also use these settings, which leak information. This all just comes down to thinking through the security scenarios. Just like developers need to consider the usage of their applications or APIs or frameworks from the user perspective, those tasked with securing up their applications need to consider how potential hackers might try to access key information to help them hack your systems.

Identifying the framework or "stack"

Let us check if your application smells too much like you are using a particular framework or another. This may or may not be terrible, BUT if you can tell which version of the framework you are using from publicly accessible information on your site, then you are leaking too much information. In fact, some firms/developers advertise very publicly which frameworks or stacks they use, but they usually try not to talk too much about the exact versions they use. For example, if an application responds with a 200 OK HTTP status for the majority of the following URLs, it is VERY likely it is a Rails application:

/javascripts/application.js
/404.html
/500.html
/422.html

In the case of the last URL, we can identify that the application is written using Rails 2.0. We can also look at the 404 to see if it looks like the default looking 404 file for different versions of Rails. If /422.html doesn't exist we might still be able to tell the difference between Rails 1.1 and 1.2 applications by what /404.html looks like. Moral of this story: At a minimum always change the default error pages in your Rails application to fit your design. Also consider changing the URLs of the error pages too. Not only does the redesign prevent potential hackers from identifying which version of Rails you might be using it also looks more professional and makes the user experience in case of an error a little more acceptable! If you are not using Rails, think about what files are created by default by your framework and change things around a little to prevent information leakage. I will continue this series in subsequent parts. Stay tuned!

Stupid Easy: 7 Deadly [Rails] Sins, Part 3

2008-01-31T09:23:00.000-06:00

This is the third and final part in the Stupid Easy: 7 Deadly [Rails] Sins series. See part 1 and part 2 if you haven't already. I have left the most heinous sin to the end. If you are a serious sinner in this aspect, there is little hope for you unless you take steps now to rectify your behavior in this aspect.

Stupid Easy Ruby

When in Rome, do as the Romans do

That's right. This is the sin of not knowing how to really code in Ruby. Sure some coders that move to Ruby can write code that does what they want it to do, but often at the severe expense of compromising project code maintainability (e.g. making things very unDRY, etc.) or even introducing significant performance issues by not understanding basics about the Ruby language. Instead I see people using Java, Perl, PHP, C# or even Python idioms in Ruby instead of understanding the essence of the language of Ruby, which is distinct from each of these languages in different ways. The same is true for any language/environment you work within. I know former PHP people that moved to Java and do not have the first clue about basic Java idioms that work well. This isn't just to pick on PHP heads as I imagine this occurs from any language migration path. I have simply seen this sin committed most by former PHP coders moving toward Java, Ruby and Python. However, I will say I see a LOT of former Java (massive static language proponents - Joshua Bloch you are a nonobjective snob) developers that do not understand or appreciate the non-static design mentality and they create large class hierarchies and don't understand what a Mixin is (or they pretend to be "cool" and talk alot of about these idioms, but don't really utilize them in the right way themselves). The Ruby mindset is still different from it's more similar looking cousins: Python and Perl, but moving between Python, Perl and Ruby (in any direction) feels more intuitive (at least in my mind). Now I should stress that I can no longer code in Java without pulling my hair out because the thought of creating 5 interfaces and a factory for every three class implementations drives me temporarily insane. The SPI design principle allows Java to be fairly flexible (especially for a statically typed language), but at the expense of my personal sanity now that I prefer to think the non-static way (yes, I still appreciate Python's strongly-typed ways). This is why I prefer to use JRuby, Jython or Groovy scripting in Java environments. There is no magic bullet to stop sinning in this regard if you aren't willing to take a journey to learn and understand Ruby idioms (not just basic *knowledge*). There are no shortcuts, it simply depends on how easily your brain can shift in gears. This doesn't mean people who can't shift gears are "bad" developers, but not well suited to migrating to significantly different environments very often. Just remember the rules in Ruby aren't the same. Think in terms of sending messages to objects that may or may not respond to the message you send, instead of expected interfaces and you are half way there if you are coming from Java, C++ and similar languages. Think of Java as a big government structure with a complicated tax code and Ruby as minimal government with a simplified tax code without loopholes accessible only to the rich! That means in the Ruby world wild things such as adult services can legally be procured between consenting adults, so bend your brain to think that way when working in Ruby...almost anything is possible especially in the realm of metaprogramming that is not possible in the Java world.

That government is best which governs least.
--Thomas Paine

I personally happen to agree with Thomas Paine's statement (at least in the context of programming), but there are various programming philosophies that different languages cater for, so it comes down to your own preference. Pick the ones you can live with at the time and think in terms of and you will be happier for it. Obviously in a programming context some people (e.g. Joshua Bloch, who is still a static language snob) will oppose legal procurement of "adult services". They have a moral objection to these activities and thus do not appreciate the beauty of dynamic design. If you have such moral reservations in programming, do everyone a favor and stay in the Java world or at least static languages! This is the beauty of different opinions. Nobody is right in absolute terms, but you need to make the right choice for yourself. It is the zealots who only accept "absolute correctness" that we should be weary of (on any side of the argument, but including Joshua Bloch). My general philosophy when I had the Java Way ingrained in my brain was that as an API designer I had to protect stupid developers from their own stupidity. Now my philosophy in the Ruby world is that I write APIs for smart developers and if you aren't [Ruby] smart, use at your own risk. Of course, I oppose complication for the sake of appearing to be "smart". But I try to write APIs that allow more advanced Ruby developers to take advantage of language features such as blocks, metaprogramming, etc. as opposed to providing an over-simplified, dumbed-down API that ends up looking ugly to the wiser Ruby developer due to being less DRY than might be possible without restriction.

Ruby Language Pointers

If you haven't acquainted yourself with the pickaxe book you should pay special attention to the following sections: Blocks, Mixins, Inspecting Objects, Inheritance & Messages, Inheritance & Mixins, Classes & Objects Interacting, The Ruby Language. Then I would look at the source code of the ActiveSupport open source project (part of Rails) for techniques that are very useful. Some of these techniques I have discussed in previous blog posts: Ruby Idiom: Reopening Classes, Ruby Idiom: forwardables, Ruby Idiom: Inject-ing Understanding, Ruby Equality, Ruby Higher Order Messaging.

Stupid Easy: 7 Deadly [Rails] Sins, Part 2

2008-01-22T09:43:00.000-06:00

In the second part of Stupid Easy: 7 Deadly [Rails] Sins, we look at sins four through six and save the most heinous sin for last in the third part of this series.

4. Stupid Easy Views

Everyone has seen spaghetti view code, either in the JSP, PHP, or other similar web templating environments. So why do people honestly think (ok, only the no-thinkers) that when they use Rails that you can't violate MVC because it is built around the MVC "pattern". Don't even get me started on the whole "pattern" thing... If you see a piece of view code like the following, then you are violating MVC. Yes, even when using Rails:


<% 
if current_user && current_user.admin?
  @somevar = something goes here
  # and do something else here
else
  @somevar = something else goes here
end
%>

This is a small (but very stupid) violation, but a violation none-the-less. I have seen not just logic in view code, but oddly model-specific code (in one case I even saw some attributes of a model being updated at the END of the view template - go figure). Now simply checking the admin flag of the current_user to insert some view specific code, would not be violating MVC. There are various ways of making sure you don't make this mistake. Some people think using a specialized templating language in place of eRB is the way to go (one of the many potential replacements that I happen to know about is Haml). These specialized template languages basically make it virtually impossible to embed any kind of logic into the view code. Others (pro-thinkers, rather than no-thinkers) don't believe that level of enforcement is necessary and just make a point of highlighting to the team (and then quickly refactoring) any violations in view code to educate all the team members about this sin. I have even seen some Rake tasks written (ok, I may have written a couple) that flag any views that have specific keywords within the <% %> eRB brackets as potential violators so that we can look through the suspects and make judgments before a release is made (because you never want to release code that violate basic principles you believe in). Another strategy is to only work with developers that you share a very similar development philosophy and know they wouldn't violate this basic rule of thumb in the first place. In any case, you need to incorporate one of these strategies so you can be sure to be rid of the Stupid Easy View faux pas in Rails projects.

5. Stupid Easy Configuration (and Routes)

In Rails 2.0 there is absolutely ZERO excuse for this (with very nice initializer hooks available), but even pre-2.0 it is hard to justify an config/environment.rb consisting of more than 30 lines (including basic Rails configuration minus comments). Separation of configuration into separate files that are required by config/environment.rb were my personal preference in pre-2.0. On the config/routes.rb side, I would often see the default routes still uncommented. In Rails post-1.2 there is really no excuse since almost everyone should be using RESTful routes with RESTful controllers (even if the way Rails generates a ridiculous amount of code, when it could be refactored nicely into a mixin or base class, is pretty dumb) anyway. In fact, default routes are almost always an invitation for troublemakers to hack into your site (especially if they know it is a Rails site). I can't say I have never deployed a Rails 1.2.x site with default routes uncommented, but I certainly haven't in the last year since Rails 1.2's official release.

6. Stupid Easy Library Code

For some reason numerous popular Rails plugins promote the idea of putting generated files into your RAILS_ROOT/lib directory (e.g. acts_as_authentication, restful_authentication, etc.). Instead of being a real plugin they are basically just Rails generators. Now we could debate how generators could be better supported in Rails, but that is really a tangent to the point. Don't be fooled by these popular "plugins" (remember they are really only generators). If you have significant code in your RAILS_ROOT/lib that has the same focus (e.g. authentication, authorization, credit card processing, etc.), then you really ought to create your own plugin (even if it isn't going to be shared with other projects in the foreseeable future). How you and/or your team defines "significant" is a judgment call, but I personally think any unit of code that works together that is greater than 200 lines of elegant Ruby code is significant (minus empty lines and comments of course). It isn't necessarily a hard or fast rule, but a general rule of thumb, which may have exceptions. The main purpose of this is twofold from my perspective. When you have a lot of code in your RAILS_ROOT/lib it is a harder to be more disciplined about organization. Whereas with a plugin you know where plugin initialization code goes vs. mixin or other code. Another big benefit (which is related to code organization) is testing/specing. It is much easier to see (or specifically notice) test/spec-coverage shortcomings of plugins than holes in your tests/specs for RAILS_ROOT/lib code. Benefits of creating plugins include that it is forces you (the developer) to think more about:

Testing/specing your code
code organization from a maintainability perspective
what chunks of code go together (should this be one or two plugins)
the scope of your plugin vs. what needs to be defined or written in your application code
who is responsible for developing this functionality, maybe it is the infrastructure/core/framework development team who needs to be responsible for authentication or authorization code rather than application developers. With plugins you can separate our these responsibilities very easily

In addition, a plugin is much more easily shareable across projects (in a more organized and disciplined way than just copying or even linking files/directories into your RAILS_ROOT/lib directory). Of course, I also find the way Rails plugins are supported by Rails a hindrance, especially when RubyGems could provide a many facilities to support this in a more elegant way, but that is a topic for another day!:) There is only one Stupid Easy Deadly [Rails] Sin left, which I will publish in the next two weeks.

Stupid Easy: 7 Deadly [Rails] Sins, Part 1

2008-01-12T10:46:00.000-06:00

Everyone has been talking about Rails for a few of years now as the framework that makes it "stupid easy" to create web applications. The problem I have with this mentality is it creates a "stupid easy" sub-culture within the Rails community that promotes stupid laziness (as opposed to smart laziness). Over the last 3 Rails contracts, I have worked with supposedly experienced developers that have switched to Rails from OO languages like Java and C++ (as opposed to pretend OO languages like PHP). The results have been VERY disappointing. The biggest problem I see is that some, even quite experienced developers, have the [stupid] lazy mentality burned into their brains. They want to visit blogs and copy and paste code into their applications without thinking about it. In fact, recently I found a ridiculous real world example of such madness. No thought put into the pasted code AT ALL. I could identify the blog posting on a popular Rails blog that had the EXACT same code. The variables weren't EVEN renamed to be meaningful in the current application. Instead a finance web application had references to customers and invoices in the controller and view code when there were no such entities/models involved in the application. Not only is this a maintenance nightmare, it show how little thought went into the code just to get a small AJAX effect that (a) wasn't difficult to write from scratch (4 lines in total after refactoring from the 11 lines of code used) and (b) was probably not even that necessary from the user experience perspective. These are the developers I think ought to return to their no-thinking PHP or Java/JEE recipe books and leave the Ruby world alone. Of course, the authors of "recipe" books should also bare some of the blame for encouraging the no-thinking hackers out there to join the Ruby world in the first place. Is that really the type of mindset we wish to harbor in our community? Look at the PHP and Java universes today for a reference point of where we will end up very soon if we are not careful.

1. Stupid Easy Schemas

One of the deadly sins I saw on my travels through their code was extending the "stupid easy" mentality in the form of not defining schema properly. Sure they *used* Rails' migrations, but they didn't really create a usable schema based on the use stories and cross-story functional requirements. While you might not want to use database specific features like triggers, stored procedures, or even foreign keys (the latter assumes you are using Rails-based equivalents in place of FKs), you should still make your schema sensible. For example,


# Migration code
  create_table :countries do |t|
    t.column :code, :string
    t.column :name, :string
  end

# Model code
class Country < ActiveRecord::Base
end

In this example we created a table called countries that has two string fields: code, name. In this case the code is the ISO country code that the application is going to use for almost all country lookups in the database. We knew that pretty much from the beginning because of the nature of our application. (Note: this example is a little contrived as I had to change things so as not to expose too much about the application this is from). Not only do we know that the Country model will almost always be looked up by its code (from the initial set of use(r) cases/stories we need to implement). The ISO code is also always 3 characters long. We also know that these ISO codes are unique for each country. All these things we knew before we needed to create the schema. This is the important information that is missing from the schema and the model is missing relevant validation. Before committing this new model and migration for Country the developer writing the code (one of the developers I am thinking of loves to tout how XP/agile and great his code is), should have had the following code on initial checkin (there really is no excuse IMHO):


# migration code
  create_table :countries do |t|
    t.column :code, :string, :limit => 3
    t.column :name, :string # You may even decide to cap the length of the name field too, but this is more of a DBA style thing.
  end
  add_index :countries, :code, :unique => true

# Model code
class Country < ActiveRecord::Base
  validates_length_of :code, :is => 3
  validates_uniqueness_of :code
end

While on my soapbox about this point, another developer I respect said this might be premature optimization. In this specific case, I totally disagree, however, this is a good point to make in general. You don't want to do premature optimization either. This is something we all need to be careful about and to be aware. This is the point where you need to think for yourself. There is no cheat sheet on these types of thinking points. The [stupid] lazy Rails developers should go back to hacking PHP senselessly or following J2EE/JEE blueprint patterns if they find thinking for themselves to develop their own rules of thumb too much work. Remember [smart] lazy is the way to go.

2. Stupid Easy Models

Another cardinal sin I often see in both client work I have inherited from others or open source projects is dumbing down models. Instead of creating member methods of models that related to *what* they are and do, some Rails coders (usually not very experienced) write this functionality within the controller layer, which leads to drastic controller layer bloat. Which is really ugly. Remember if your code is starting to look ugly, do something about it. The way to fix this is *almost* always to create relevant member methods on the corresponding models that this functionality works on. One easy example of this might be to add an authenticate method to the User model instead of having extra logic in the ApplicationController. Think CRC instead of procedural.

3. Stupid Easy Controller Methods

This is perhaps the most obviously harmful (from a security perspective) sin. While traveling through the Supid Easy Rails universe of code, I almost always see unprotected controller methods that are used as filters or for utility purposes in some capacity. This is a sin that I myself must confess to, though I haven't done so since the first 3 months of my Rails development around 2005 Q4. The following is typically what I see around:


# in SparksController
  def check_permissions
    return false unless current_user.has_permission?(:MANAGE_USERS)
    true
  end

The problem with this is that we can do the following (unless using resource routes where default routes are taken out):

GET /sparks/check_permissions?some_var=some_value

And perhaps hack into the web application. In this case we may not be able to do anything too interesting, but there are other scenarios that this could create a very open hole in the web application. The way to fix this and keep your team's sanity from a maintenance perspective is to protect these filter or utility methods by scoping appropriate with protected or private, which is not difficult at all. However, if you are a true Rubyist you will probably opt to keep your controller utility methods in Acts::As mixins or similar.

'Twas the night before launch...

2008-01-04T19:10:00.000-06:00

Oh fun times! On such fateful nights you know you will find something that will screw things up unless immediately addressed. It must be someone's law already, but if it isn't I call dibs on it and it should thus be called "Potter's Law":). On a night before launch that I recently experienced, I had a heart stopping moment. The last 5 days of the sprint I had been churning out last minute changes based on usability feedback from the client. In the process, I had neglected to include benchmarking and memory leak testing with the same discipline I prefer. On the night before launch I ran my usual benchmarking scripts. The results were pretty good except on one action. We could have lived with that performance, but the real problem came when I ran tests to detect memory leaks in my Rails application. Let the real fun begin! Before continuing, I should mention there is only thing I hate more than uber visual tasks (editing/creating graphics, layout tweaking, etc.) and that is debugging memory leaks. I ran the ab (Apache Bench) utility on a few different types of pages in the staging environment and didn't notice any problems. The Rails application was consistently teetering under 50M RAM. Excellent! Then I tried the second most requested type of page (this was a static site we were rewriting in Rails to create an easy to use domain-specific CMS - so we had live production web statistics) and my heart skipped a beat or three. Memory usage spiraled out of control. From 49.5M the single mongrel process eventually grew to 250M after a few ab -c2 -n100 .... runs. I first checked out the view since I knew the controller action code was only 6 lines (how much could go wrong there?). I tried removing different parts of the view code and rerunning my tests while monitoring the memory usage of the process. Still no change. So I reluctantly looked in the controller and saw something like the following:


  def show
    @eto = ExchangeTradedOption.find_active(params[:id])

    respond_to do |format|
      format.html # show.rhtml
      format.xml  { render :action => "xml", :layout => false }
      format.csv  { render :action => "csv", :layout => false }
    end
  end

What on earth was in the ExchangeTradedOption.find_active method?


  class << self
    def find_active(id)
      now = Time.now
      find(
        id,
        :include => [:exchange, {:vendor_symbols => [:vendor]}],
        :conditions => [
          %{options.expires_at <= ? AND options.active = 1 AND 
            vendor_symbols.effective_date <= ? AND vendor_symbols.expiration_date >= ?}, 
          now, now, now
        ],
        :order => 'vendor_symbols.effective_date DESC'
      )
    end
  end

Ouch! The problem was that I still needed all vendor symbols and vendors for the view in question and didn't want to make the extra SQL queries for the Vendor on each VendorSymbol as that would have added between 5-10 extra SQL queries per action invocation. I guessed the nested includes in the find was most likely to be causing the issue. So I refactored as followed:


# Controller action code: ExchangeTradedOptionsController#show
  def show
    @eto = ExchangeTradedOption.find_active(params[:id])
    @vendor_symbols = VendorSymbol.find_active_for(params[:id])

    respond_to do |format|
      format.html # show.rhtml
      format.xml  { render :action => "xml", :layout => false }
      format.csv  { render :action => "csv", :layout => false }
    end
  end

# Model finder code: ExchangeTradedOption.find_active
  class << self
    def find_active(id)
      now = Time.now
      find(
        id,
        :include => [:exchange],
        :conditions => [%{options.expires_at <= ? AND options.active = 1}, now],
      )
    end
  end

# The VendorSymbol.find_active_for method implementation is left as an exercise for the reader
# but it is very, very simple!

# Changed view to refer to @vendor_symbols array instead of @eto.vendor_symbols
# This also helped reduce indirection and prevented Law of Demeter violations in the view code!

Of course, I didn't leave it without verifying! Never just guess that the problem is solved, you should always verify this with tests (either automated or manual). There were a few solutions, of course. One is the one I provided above. The second solution I thought of trying was removing the vendor_symbols SQL conditions and sorting in Ruby. This seemed a waste of CPU to me and extra lines of code I felt was unnecessary (remember the less code you write the less you need to maintain!). The third solution was to execute an optimized raw SQL query myself. I also didn't like this option as the first one presented about seemed cleaner and more maintainable going forward. The first solution does cause more than one SQL query to be executed each time the action is run, but it is scalable as it remains constant at two queries per invocation. In addition I added one extra instance variable, which is also not ideal, but again this didn't seem to me to be the worst problem at this stage with only two instance variables in the action being set for the view. The benchmarking results proved my assumptions correct regarding the various solutions. The moral of the story is find the best solution based on the context rather than attempting to apply a one-query-per-action-invocation rule to the whole application, which might be ideal, but occasionally unrealistic and/or catastrophic.

2008 Predictions

2007-12-30T14:59:00.000-06:00

Since the end of the year is almost upon us close associates of mine have been asking my predictions for the new year. While obliging them here I suspect in 12 months time I will review this blog posting in (most likely) absolute humiliation and horror as most that make predictions do.

Web 2.0 will die in 2008 and Web 3.0 will reach adolescence in comical fashion, but will not receive the same private equity interest of its predecessor.
Carbonless Datacenter will be the IT buzz phrase for a good part of 2008.
SaaS will morph into yet another acronym (hopefully one that makes more sense in terms of capitalization) that will take into account yet another business model twist.
DHH will continue to be perceived by many inside and outside the Ruby/Rails world as an arrogant a$$.

Lastly, I predict that Tony Blair will concede and sincerely apologize for his disastrous 2002 prediction that Weapons of Mass Destruction (WMD) exist in Iraq that has already caused a much higher death toll (that continues to mount) orders of magnitude higher than the unmistakable military fiasco of the Charge of the Light Brigade. Sorry I forgot Mr Blair doesn't have a sincere atom in his body and this is only a tech prediction list.

Don't outsource your thinking process. Think for yourself!

2007-12-06T22:54:00.000-06:00

In the wirlwind of political turmoil and occasional nuance of late with the upcoming primary elections almost upon us, I realized just how offensive the two-party primary elections ought to be to a "free" country. Why isn't everyone else as outraged as I am? Traversing various circles I see Democrats hashing out their own political differences on foreign policy, social security and the overloaded and antiquated driver's license system to create a unified platform for the general election. As if *ALL* Democrats think the exact same way. Republicans on the other hand are dumb-founded and depressed by the lack of quality mainstream and viable candidate(s), not to mention trying to figure out what on earth their united platform is after a president that has probably done more to harm their party than any other individual in the last 25 years (hey, don't look at me or the Democrats, neither of us voted for him in the first place!:)). Oh, I do enjoy being smug! Being a foreigner I always found it odd that Americans (generally) seem more willing to outsource their political reasoning to one of two political parties (at least in contrast with my European experiences). I have witnessed the deferral of political reasoning to their parties without thinking on so many occasions I have lost count. If their *party* is anti-war, then they need to be too, no questions asked. Alternatively if they are with the *other party* they need to buy into the whole Cheney-masterminded war on terror doctrine, where we must "take the war to the enemy", whatever that means. Now back to the technical world (and this transcends borders). I see the same type of decision/reasoning outsourcing. In 1997, Apple coined the slogan "Think Different". Fast forward 10 years and you have an ever growing Apple fan base that doesn't question (at least not Apple or Jobs), it just doesn't "think". I am not saying Apple hasn't done anything good, but all you Apple fan boys (and girls) just think for yourselves once in a while, puh-lease! To be fair and candid, it is not limited to the Apple fan base. The Linux community (of which I am party) has been a huge propaganda machine for just as long. A friend of mine who is primarily a Windoze user (sorry, I just couldn't resist) suggests that the Linux community has been pushing Linux (or at least certain distributions of it) as "just as usable out of the box as Windows", which he thinks is a sad and pathetic misrepresentation. To be honest, I agree with him. I don't think any Linux distribution is quite as end-user friendly as Windows out of the box and to me Linux is definitely NOT about that. The Ubuntu weekender geeks that desperately try to be the geekiest person in the room should take note (among others). Forget pushing Linux propaganda on others (remember we don't like Jobs-indoctrinated fan boys/girls selling us the Apple way either). Accept Linux for what it is: a more exclusive club that sets the bar higher than the others! Why should we be trying to lower the bar or grade? Let the newbies reach for the higher bar to get started, so we can attract a higher quality user base than the other two options. The BSD community is a great example! I also feel this way on the Ruby front. I do wish all those nuts-oh "Rails rulz" propagandists would steer clear of all the PHP and Java bashing and Rails hyping. Do you really want all the subpar PHP and Java people (the ones that need hype so they can outsource their decision making to move over) in our world? I personally only want to cherry pick the Java, PHP, Python, etc. developers that actually think for themselves and willingly and enthusiastically step over to the Ruby side. In summary, if you want someone else to think on your behalf go ahead it is a "free" country in that respect (if you aren't in a free country remember nobody owns your internal thoughts so you may as well abuse that freedom as much as you can:)!). However, you should also realize that it is only the free-thinkers that don't conform to mass expectations and opinions that are the ones changing things (mostly for the better, but feel free to disagree with me and form your own opinion:)).

My .gitconfig

2007-11-10T07:29:00.004-06:00


[user]
  name = Susan Potter # make sure you change this
  email = me@susanpotter.net # make sure you change this
[color]
  diff = auto
  status = auto
  branch = auto
[diff]
  rename = copy
  color = auto
[apply]
  whitespace = strip
[pager]
  color = true
[status]
  color = auto
[alias]
  co = "checkout"
  ci = "commit"
  ciall = "commit -a -v"
  unmerge = "reset --hard ORIG_HEAD"
  lsbr = "branch -a" # list all branches, even remote ones
  mkbr = "branch" # create branch if you specify a branch name after it, e.g. git mkbr upgrading_rails
  # remove branch named after it, e.g. git rmbr upgrading_rails
  rmbr = "branch -d"
  # rename branch from one name to another
  mvbr = "branch -m"
  # 
  track = "branch --track"
  # list all tags, to keep commands consistent, e.g. git lstag
  lstag = "tag -l"
  # create a new tag based on specified commit
  mktag = "tag -a"
  # remove existing tag by name
  rmtag = "tag -d"
  # rename tag from one name to another
  mvtag = "tag -m"
  # create new remote repository for project
  mkrem = "remote add"
  # list remote repositories
  lsrem = "remote"
  # show status, keep same as svn command I used most frequently
  st = "status"
  # another alias for status that some scripts might use
  stat = "status"
  # fetch and rebase from svn repository
  spull = !git svn fetch && git svn rebase
  # push keeping each local commit as atomic.
  spush = !git svn dcommit
  # initialize all submodules
  modinit = "submodule init"
  # update all submodules
  modup = "submodule update"
  # show status of all submodules
  modst = "submodule status"
  # add new submodule, i.e. git modadd module-name url
  modadd = "submodule add"
  # show last 15 log entries
  recentlog = "log -n 15"
  # push local committed changes to rubyforge and origin (usually GitHub)
  osspush = !git push rubyforge master && git push origin master
  # pull changes from rubyforge and origin (usually GitHub)
  osspull = !git pull rubyforge master && git pull origin master
  # sync (pull then push) from rubyforge and origin (usually GitHub)
  osssync = !git osspull && git osspush

Place your .gitconfig file in your home directory, e.g. /home/username Last updated: 2008-06-28

Twitter4R v0.3.0

2007-11-05T02:18:00.000-06:00

After Twitter.com added a few extra APIs for favoriting statuses in October, I finally got around to adding them to Twitter4R tonight. Since there is new functionality I have released it as version 0.3.0. To install all you need to do is:

$ sudo gem install twitter4r

The Rubyforge mirrors might still be syncing if you are doing this soon after I post this message. The changes to the library include:

Added Twitter::Client#authenticate? method that is a lightweight way to verify a Twitter user's credentials
Favorites: allows Twitterers to add statuses to their favorites, list all their current favorites and remove any statuses from the list.

Example code for verifying a user would look like:

gem('twitter4r', '>=0.3.0')
require('twitter')

client = Twitter::Client.new
unless client.authenticate?("macdeveloperthatlovesleopard", "iamasciolists")
  puts "Password does not describe user well!"
end

if client.authenticate?("developerthatdoesnotpretendtobegraphicdesigner", "seriousdevelopernotfakestevejobswannabe")
  puts "Password does describe user well!"
end

Example code for using the favorites API:

gem('twitter4r', '>=0.3.0')
require('twitter')
require('twitter/console')

client = Twitter::Client.from_config('some/path.yml')
statuses = client.favorites
ids = statuses.collect {|s| s.id }
# clean favorites list by deleting
statuses.each {|s| client.favorite(:remove, s) }

# now add back first status id from original favorites list
client.favorite(:add, ids[0])

Enjoy!

The Democratic debate, Ann Coulter and cryptographic utilities for Ruby application developers

2007-10-30T23:11:00.000-05:00

An unlikely trio, but... After watching the flip flop of Clinton in the Democratic debate, the subsequent carcass ravaging by her competition, UFO spotting by spock (aka Kucinich) and two great stinging one-liners from Joe Biden about the presidential [dis]qualification of candidate 911, I stumbled across the infinitely dim and unimpressive Ann Coutler. She was attempting to justify yet another deliberately outlandish statement she made last week, a desperate bid to remain in the limelight to which she is so addicted. At this point, I remembered I wrote a RubyGem a few weekends ago that might be able to decrypt these events and make sense of it all (well maybe!?). A few weekends ago I released metafusion-crypto as a Ruby Gem. I hadn't announced it yet as it got lost in the shuffle of some personal matters that forced me to take five days off work unexpectedly. So here is the belated introduction to that small, humble Gem. This metafusion sub-project is basically just two utility classes: Metafusion::Crypo::PrivateKey and Metafusion::Crypo::DigitalSignature for application developers that don't want to worry about twiddling bits. To install you only need to do the following:

$ sudo gem install metafusion-crypto

An example of DigitalSignature usage might be:


require 'metafusion/crypto'

priv_key = Metafusion::Crypto.generate_key_pair
# this returns OpenSSL::PKey::RSA object into priv_key

include Metafusion::Crypto
sig = DigitalSignature.from_keys('rsa_key.pub', 'rsa_key')
original_text = "my clear text message"
crypted_text = sig.encrypt(original_text)
plain_text = sig.decrypt(crypted_text)
puts "It worked - let's celebrate!" if original_text == plain_text

An example of PrivateKey usage might be:


include Metafusion::Crypto
pkey = PrivateKey.new('mypassphrase')
original_text = 'Yo yo yo.  What up dog?'
crypted_text = pkey.encrypt(original_text)
plain_text = pkey.decrypt(crypted_text)
puts "Let's roll and celebrate - it worked" if original_text == plain_text

I look forward to application developer's feedback. I promise to return to the agile anti-pattern "series" soon. Update: Since two people asked, I have stuck to my tradition of releasing with only 100% C0 code coverage of the project's RSpec examples.

Agile Anti-Patterns, Part 1

2007-10-18T21:16:00.000-05:00

In the last 12 months I had the fortune to work at a client that calls themselves "agile". In fact they use marketing blurbs similar to "on the cutting edge of agile Ruby on Rails development" (I said *similar* because I didn't want to identify them uniquely). The problem is when I briefly worked on a project for them, the truth was very far from their marketing spiel. At a minimum I can say that this was the case on the particular project I briefly worked on, but more likely to have been much more wide spread in the firm since the consulting manager managed all the client projects from what I could tell. Why was I fortunate? Well first of all, it was a very brief encounter. I just couldn't bring myself to work in that environment for long (and thus you will not find it on my resume at all - nice try). Perhaps that says something about me too (I know it does - it is a liability, I realize this and plan to work on it, but on the other hand I feel as if I shouldn't have to compromise my professional values to the level of this case). Secondly, like I imagine is the case with most people, I learn more from bad experiences than good experiences. I would also like to say this is by no means the only company that likely falls into this "let's talk about agile so much we pretend that in itself is being agile" trap. I see 1,000s of job ads or consulting firm spiels saying agile-this, agile-that. Probably only 10-20% of the spiels are decently aligned with reality (of course, I have no way of knowing or measuring this, but it seems like a sensible guess to me;)). Now, is any agile team perfect? No. At least not in my experience. Maybe there are perfect teams and I am just a loser that has never experienced one, but since I have never met a perfect person ever, I imagine this to be improbable. The reason for my deduction is that since software agility is based on people and relationships NOT just tools, then only a team with perfect members could even dream of producing the perfect dynamic. The thing is, you don't need an agile team to be perfect. That is the premise of each agile practice I follow personally. If you think about it, agile practices assume people at their core are faulty some of the time in some way, because we are mere mortals, not software gods (though at times developers do get a God-complex, which is one of our many faults - I am guilty here too)! Ooops, tangent, but is serves a point here too. I do not ever wish to imply that following the letter of all the agile practice "rules" will make you truly and *perfectly* agile. I do not even think that is even the point. For me it comes down to Kaizen, which in short means, to iteratively eliminate waste from a process each time you repeat and to break those iterations down into bite size pieces so you can track and measure progress (actually that is a very coarse way to describe Kaizen, which is a word I love so much I tattooed it on my body somewhere - just kidding.....or am I?). Feel free to search the web for a better description (there are a few), but it is a word the more you read about (at least for me) the more you appreciate. Perhaps we will never be perfect agile developers. Perhaps we will never be perfect business analysts, coders, managers, deployers, system administrators, database administrators, etc. That to me is almost irrelevant. The idea is we just need to strive to simply "do better" each time and have the right attitude. This was really just the kick off piece (aka rant) for this "series" on anti-patterns relating to agile practices. I'll have concrete examples coming soon in part 2, 3, etc. Stay tuned. PS If you are searching for an "agile" development post, don't despair, there are some decently agile places out there. They may not be perfect, but hopefully better than the client project I will discuss.

Coming out....in support of DHH [to some extent]

2007-10-11T21:16:00.000-05:00

Since today is National Coming Out Day in the US I thought I would use it somehow in my blog entry. Yes folks I am "coming out" in support of [some of] DHH's decisions. Earlier today I noticed a lot of blog chatter regarding Rails and what various people think it *should* or should not be. While I agree with some of these people's sentiments that DHH is an arrogant ass, I also think he has made *some* smart decisions (though on a number of mid/lower-level issues I think he has it totally wrong). Now reread this paragraph until you get the point that I still think he is an arrogant ass with a God complex. Firstly, let's discuss the "Twitter" issue. Yes, I use Twitter (occasionally), though I see it's biggest impact (for my personal interests) in marketing rather than tweeting every time I use the bathroom (note: I am not part of the look-at-me-NOW-Gawd-damn-it generation - I was a humble 70s baby). In fact, the marketing potential is what drove me to develop Twitter4R in the first place. What I don't like about Twitter is due to their own lack of foresight they desperately tried to blame a framework! An application framework because they couldn't get their act together to do the necessary scalability testing of their whole deployment environment (which includes many things, only one of them is the application framework) before hand. This is the same application framework that got them rolled out into production earlier than they would have with other frameworks. The same framework that allows them NOW to scale to crazy amounts of hits per second! I attempted to follow the dialog between the Twitter developers and Rails Core team, though not too successfully because it occurred through non-interlinked blog posts and comments. However, from what I could gather the Twitter folk seemed to think it should be up to the application framework to create an already baked in way for them to setup replication models. If they were using a PHP framework, they wouldn't have bothered asking since all the DB code is in the view anyway! Why is that the responsibility of the application framework? Now if Rails has somehow prevented them from being able to create ActiveRecord extensions to do this, I might understand, but as far as I can see....this just isn't so. Secondly, the CDBaby rant. If CDBaby took 2 years to release using Ruby/Rails, then they must have had much bigger problems than Rails itself! Sorry to be blunt, but who takes that long anymore, even in a Java stack? Ok, now for some leveling. I still think DHH is an ass and an extremely arrogant ass at that. I still think he has ridiculous ideas about trademarking logos (that he didn't even pay for), definitions of what components are, how vendor/* is better than utilizing Ruby Gems in virtualized or dedicated environments (WTF?) and general dependency loading in Rails. Plus, did I mention I think he is an arrogant ass? Not to mention a former PHP programmer...enough said! Good night.

Twitter4R Announcements

2007-09-23T20:46:00.000-05:00

First off I have been meaning to post an entry letting people know of a recently released Rails application written by Sergio Santos called TwitterNotes that is using the Twitter4R gem. Thanks Sergio (and anyone else that worked on the project - sorry if I left you off). Also Sergio emailed yesterday requesting message paging support with a suggested patch (thanks for contributing back - muchas gracias). This evening I released Twitter4R version 0.2.5 that incorporates the necessary code changes for this support. What does this mean? Previously you would have only been able to get the last 20 sent/received direct messages from the Twitter4R bindings doing something like the following:


messages = client.messages(:sent)

Now you can request specific pages like so:


messages = client.messages(:sent, :page => 2)

The same is true for the :received case. To install the Twitter4R RubyGem simply run the following in your terminal/console:

$ sudo gem install twitter4r

Give the Rubyforge mirrors a few hours to sync if you are only getting Twitter4R v0.2.4. Thanks and enjoy!

Custom Processors for ActiveWarehouse ETL

2007-09-21T00:45:00.000-05:00

For anyone interested in extending ActiveWarehouse ETL's features with custom pre/post processors, I thought I would share this piece of code that I wrote in August for a personal project I am working on. The example should provide you with enough details for you to create your own custom processors.


# Written by Susan Potter  under open source MIT license.
# August 12, 2007.

require 'net/ftp'

module ETL
  module Processor
    # Custom pre-processor to download files via FTP before beginning control process.
    class FtpDownloaderProcessor < ETL::Processor::Processor
      attr_reader :host
      attr_reader :port
      attr_reader :remote_dir
      attr_reader :files
      attr_reader :username
      attr_reader :local_dir
      
      # configuration options include:
      # * host - hostname or IP address of FTP server (required)
      # * port - port number for FTP server (default: 21)
      # * remote_dir - remote path on FTP server (default: /)
      # * files - list of files to download from FTP server (default: [])
      # * username - username for FTP server authentication (default: anonymous)
      # * password - password for FTP server authentication (default: nil)
      # * local_dir - local output directory to save downloaded files (default: '')
      # 
      # As an example you might write something like the following in your control process file:
      #  pre_process :ftp_downloader, {
      #    :host => 'ftp.sec.gov',
      #    :path => 'edgar/Feed/2007/QTR2',
      #    :files => ['20070402.nc.tar.gz', '20070403.nc.tar.gz', '20070404.nc.tar.gz', 
      #               '20070405.nc.tar.gz', '20070406.nc.tar.gz'],
      #    :local_dir => '/data/sec/2007/04',
      #  }
      # The above example will anonymously download via FTP the first week's worth of SEC filing feed data
      # from the second quarter of 2007 and download the files to the local directory +/data/sec/2007/04+.
      def initialize(control, configuration)
        @host = configuration[:host]
        @port = configuration[:port] || 21
        @remote_dir = configuration[:remote_dir] || '/'
        @files = configuration[:files] || []
        @username = configuration[:username] || 'anonymous'
        @password = configuration[:password]
        @local_dir = configuration[:local_dir] || ''
      end
      
      def process
        Net::FTP.open(@host) do |conn|
          conn.connect(@host, @port)
          conn.login(@username, @password)
          remote_files = conn.chdir(@remote_dir)
          @files.each do |f|
            conn.gettextfile(remote_file(f), local_file(f))
          end
        end
      end
      
      private
      attr_accessor :password
      
      def local_file(name)
        File.join(@local_dir, name)
      end
      
      def remote_file(name)
        File.join(@remote_dir, name)
      end
    end
  end
end

The key things to note from this is that you are at present required to:

define all your custom processors with in the ETL::Processor module
name your custom processor class in the form XXXXProcessor
need to extend (or really just adhere to the message interface of) ETL::Processor::Processor class defined in ActiveWarehouse ETL
define initialize taking two arguments (look above for guidance)
define a process method to do what you need do before or after the control process runs (for pre and post processors respectively)

Hope this helps someone customize ActiveWarehouse more easily, since the only bad thing I have found with ActiveWarehouse is lack of documentation.

Separating Rails Layout Associations

2007-09-13T21:16:00.000-05:00

I just realized I hadn't shared this code yet on this blog. I will also be including it in one of the metafusion subprojects (coming soon). I've been using it when needed in my Rails projects on and off for the last several months.

The Problem

You have some controllers you include into your application from plugins or engines and you want to associate a particular layout to the plugin/engine controller from within your application (without changing anything in the plugin or engine - of course!). There is not good way of doing this nicely in Rails presently as each controller usually defines controller-wide layouts within its definition.

The Solution

[finsignia/paths.rb]

# See:
# * Finsignia::Paths

# Contains helper methods related to paths and resolving modules and 
# classes from paths.  Provides for helpful mixin for various applications.
module Finsignia::Paths

  def self.included(base)
    base.extend ClassMethods
  end
  
  # Contains class methods for Finsignia::Paths mixin
  module ClassMethods
    @@element_postfixes = {:model => '', :controller => 'Controller'}

    def resolve_model(path)
      resolve_element :model, path
    end
  
    def resolve_controller(path)
      resolve_element :controller, path
    end
    
    def normalize_module_name(path)
      list = path.split('_')
      list.collect do |item|
        item.capitalize
      end.join
    end
    
    # Resolves path to a type of Rails element 
    # (e.g. model, controller, etc.).
    # 
    # Path refers to 'internal' path, NOT require path:
    #  'users/users' #=> internal path
    #  'users/users_controller #=> require path
    def resolve_element(type, path)
      first, rest = path.split('/')
      mod = ObjectSpace.const_get(normalize_module_name(first))
      while rest
        first, rest = rest.split('/')

        unless rest
          mod = mod.const_get(normalize_module_name("#{first}_#{@@element_postfixes[type].downcase}"))
        else
          mod = mod.const_get(normalize_module_name(first))
        end
        # I doubt this will ever get this far as per expeceted behavior of ObjectSpace and Module...so commented it out following unreachable line.
        #raise NameError.new("#{type} constant not found for internal path #{path}") unless mod
        return mod unless rest
      end      
    end
    
    # Separating this so we can stub this method out in specifications.
    def require_controller(path)
      require("#{path}_controller") unless "test" == ENV["RAILS_ENV"]
    end
    
    # Separating this so we can stub this method out in specifications.
    def require_model(path)
      require(path) unless "test" == ENV["RAILS_ENV"]
    end
  end
end

[finsignia/layouts.rb]

# See:
# * Finsignia::Layouts
# * Finsignia::LayoutsError

# Raised when an exceptional condition arises in the Layouts 
# mapping process.
class Finsignia::LayoutsError < Exception; end

# Provides a closure and declarative way to define layout
# mappings for an application that utilize controllers, views, 
# etc. from plugins.
#
# The closure approach simulates the Rails Routing approach 
# closely, like:
#  require 'application'
#  
#  Finsignia::Layouts.map do |map|
#    map.connect 'users/sessions', 'layout_name'
#  end
#
# The declarative approach looks like the following:
#  require 'application'
#  include Finsignia
#  Layouts.map 'users/sessions', 'layout_name'
# 
module Finsignia::Layouts
  class << self
    def connect(controller_path, layout)
      require_controller(controller_path)
      # resolve controller class and call .layout(layout) on it.
      controller = resolve_controller(controller_path)
      controller.layout(layout)
    end
    
    def map(&block)
      yield self if block_given?
    end
  end

#  private
    include Finsignia::Paths    
end

So basically in a file like config/layouts.rb of our application we have something like:


Finsignia::Layouts.map do |map|
  map.connect 'users/sessions', 'session'
  map.connect 'users/users', 'users'
  map.connect 'accounts/transaction', 'account'
end

Then include the config/layouts.rb file in config/environment.rb and you have separated your concerns relatively nicely and easily. When I release metafusion-rails my plan is that instead of installing the finsignia/paths.rb and finsignia/layouts.rb in the lib directory you would be able to do something like the following at the end of your config/environment.rb file:


gem('metafusion-rails', '=MFR_VERSION')
require 'metafusion/rails'

Finsignia::Layouts.map do |map|
  map.connect 'namespace/resource', 'layout'
  # etc....
end

Alternatively you could still keep the layouts.rb file if your environment.rb is getting large and keep the layouts.rb include in your environment.rb.

Model-Conductor-Controller fix

2007-09-11T11:42:00.000-05:00

The New Bamboo blog talked about Presenters & Conductors last month. The demonstration Rails application was broken, so the enclosed link to a zip file contains the fixes I made to get the application running as expected. The offending code was in lib/action_conductor/lib/action_conductor/errors.rb:


module ActiveRecord
  class Errors
   def add_conductor_errors_for(record, mapping)
    return if record.valid?
    record.errors.each do |attribute, message|
      self.add(mapping[attribute.to_sym], message)
    end
   end
  end
end

The fixed code changed one line as so:


module ActiveRecord
  class Errors
   def add_conductor_errors_for(record, mapping)
    return if record.valid? or record.new_record?
    record.errors.each do |attribute, message|
      self.add(mapping[attribute.to_sym], message)
    end
   end
  end
end

The fixed zip file can be found at my 4shared account and also fixes the minor view errors.

Anonymous Classes: Java's Synthetic Closure, Part 2

2007-08-22T22:16:00.000-05:00

So in the last installment we saw something like the following (paraphrased) in Java:


    executeTransaction(session, new PersistenceExecutor() {
      public PersistenceExecutionResult execute() {
        return new PersistenceExecutionResult(session.save(persistableObject);
      }
    });

To accomplish the same thing, though in a more error prone manner, we might just do the following:


  session.open();
  Object obj = session.save(persistableObject);
  session.close();

Now you say, well that looks better. Well yes and no. The problem with this approach arises when the logic in between session.open() and session.close() is more involved. If we had 10 lines of code between the open/close markers and we came to refactor the logic, but omitted the call to close, our tests (if they are purely state-based tests from the TDD world) would probably not catch this omission at the level we would want to catch it. Remember the more levels you need to debug to find the issue the more time you have wasted not doing useful work on new stories or fixing defects, etc. Of course, in the Ruby world where closures are just there for the taking, we don't have to justify ugly looking code to help package reusable "wrapper" logic like above. In a Ruby project that utilizes ActiveRecord we can simply write:


# When a corporate action like merger, spinoff, etc. occurs for an issuer, usually 
# a new option has a basket underlying instead of just a set of equities, so code might 
# look like the following.
Option.transaction(equityOption, basketOption) do
  equityOption.substitute_with(basketOption)
  equityOption.expire
  basketOption.activate
end

So that I don't sound completely smug to my Java friends, I will let you decide which code snippet looks more beautiful!:) Update: One of my Java friends says there are no closures yet in Java 6, but there are talks about closures in 7 (or will it be version 96 next?), but it is all up in the air.

Anonymous Classes: Java's Synthetic Closure

2007-08-21T22:50:00.000-05:00

I've been meaning to write this blog post since late last year when I revisited the Java world after a hiatus of adventures in the Ruby and Python worlds. When I was a Java advocate (many moons ago it seems now), I really didn't appreciate the true beauty that is anonymous classes. I saw them as ugly and reducing code readability. Syntactically I still agree with that assessment, but conceptually I couldn't disagree more now. When revisiting Java last year for a client, I found myself really missing blocks from the Ruby world. The following common snippet of code in Ruby passes a block to a method:


  connection = Net::HTTP.new
  connection.use_ssl = true if ssl?
  connection.start do |connection|
    request = yield connection if block_given?
    request.basic_auth(@login, @password) if require_auth
    response = connection.request(request, body)
    handle_rest_response(response)
    response
  end

[This code is taken from the Twitter4R project and is slightly paraphrased.] What we are doing here is wrapping a block within a begin/end API construct. So the previous snippet of code is effectively the same as (and in fact this is the way the above gets executed):


  connection = Net::HTTP.new
  connection.use_ssl = true if ssl?
  connection.open
  request = yield connection if block_given?
  request.basic_auth(@login, @password) if require_auth
  response = connection.request(request, body)
  handle_rest_response(response)
  connection.close
  response

Why is the first so much better from a code maintainability and readability perspective? In my view the main benefits include (but are not limited to):

Passing in a block ensures we close the connection (as in the above code example). If we forgot to "end" the block in the first snippet we would get a SyntaxError in Ruby which would remind us to close the block (and thus connection implicitly). Which means less likely to introduce defects related to not closing connections unlike second code example.
Putting the code to be executed passed in with a block improves code readability and in my view that is always a good thing. It is more obvious to fellow developers what is going on.

Now in Java we can't simulate the code above in Java syntax...or can we? I suggest with a few more {} and () and ; we can accomplish the same as above. And if we cover our eyes with our hands and squint through the gaps in our fingers, we might be happy with the way it looks (OK, it is ugly, but this is the world of Java people!). Below is an example of a similar technique as the Ruby example above, but this time in Java and related to database transaction (specifically in Hibernate3 below):


// In PersistenceExecutionResult.java we define the following....
public class PersistenceExecutionResult {
  private Object result;
  public PersistenceExecutionResult(Object theResult) {
    result = theResult;
  }

  public Object getResult() { return result; }
  public void setResult(Object theResult) { result = theResult; }
}

// Now in PersistenceExecutor.java we define....
public interface PersistenceExecutor {
  public abstract PersistenceExecutionResult execute() throws Exception;
}

// Now in BaseDAO.java we have the following....
public class BaseDAO {
  public Long create(BaseObject persistableObject) throws Exception {
    final Session session = ....; // get session somehow - irrelevant to demonstration of synthetic closures in Java
    return (Long)executeTransaction(session, new PersistenceExecutor() {
      public PersistenceExecutionResult execute() {
        return new PersistenceExecutionResult(session.save(persistableObject);
      }
    });
  }

  // other stuff, but just adds noise to example.

  protected ExecutionResult executeTransaction(Session session, PersistenceExecutor executor)
    throws Exception {
    Transaction tx = null;
    PersistenceExecutionResult result = null;
    try  {
      tx = session.beginTransaction();
      result = executor.execute();
      tx.commit();
    } catch (Exception e) {
      if (null != tx) tx.rollback();
    } finally {
      session.close();
    }
    return result;
  }
}

Now what on earth is going on? Anyone? Well I will explain in a subsequent posting and compare it to the equivalent code without using anonymous classes to wrap begin/end API constructs. Right now I am too tired and lazy, but I will explain soon, fear not. Notes: the above works with JDK 1.4 because when I wrote this it had to be 1.4 compliant (therefore fancy generic interface definition to DAO was not possible). I have only used JDK 1.5 (aka 5) and not 6. Is there a nicer way of doing this in 6?

RESTful API & Web Service Statuses

2007-08-19T22:41:00.000-05:00

As I have browsed and read newsgroups, mailing lists, articles and even books related to RESTful APIs and Web Services very few tackle the issue of which HTTP status messages are most relevant for the communication at hand. Below is a list of the few HTTP status codes that I most commonly use in the development of my own APIs and Web Services for my clients:

404 “Resource Not Found”: Yes, you read that right. When a resource of a specific given ID does not exist in our datastore (whether due to it being an invalid ID or due to the resource having been deleted from our datastore). For example, “GET /people/2349594934896107” may not exist in our database, so what do we display? Just the bare “people/show” view with a flash message saying no person with that ID exists? I should say not – at least not in our RESTful world. However, if we are using something like acts_as_paranoid or home-grown paranoid delete mechanism and we know the resource used to exist in our datastore, we could respond with 410 “Gone”. The benefit of this is that we are communicating something useful back to the RESTful consumer, whether that is an end-user or another application or web service.
401 “Unauthorized”: Returned when consumer either did not provide credentials to view restricted resource or the authentication and/or authorization failed. This assumes your application uses the Basic or Digest HTTP Authentication scheme as according to the HTTP protocol the server needs to respond with specific HTTP headers in various circumstances of returning this status code.
503 “Service Unavailable”: Used when taking the site down for maintenance. I find this very handy indeed when upgrading the RESTful web services I deploy at work. For my projects I create a stub Rails application that responds with a 503 for each valid type of request that comes in. Clients of my services are usually services themselves or other applications, so this helps client developers that consume my web services know that this is a temporary blip and should be due to schedule maintenance (and a good reminder for them to check the emails I sent them over the weekend instead of ignoring them). The best and easiest way to do this is to create a new routes.rb file that contains all valid requests and points it to a controller/action that responds with a 503.
201 “Created”: Very handy, especially in the context of designing RESTful web services. Status should be sent when replying to a POST that created a new resource on the server.
202 “Accepted”: Yes, slightly obscure, but very handy. Signals to consumer of application (whether end-user human or REST client application) that a long running/executing job has been submitted for processing, but we do not yet know the outcome of the “job” or “process”. Of course, as with all of the above, to an end-user we would render a nicely designed web page telling them this, but if for some reason a bot (authorized or not) or web services client accessed our application, they would be able to determine some useful information from the status.

Please let me know any other HTTP status codes you find most useful when designing RESTful APIs. Update [2007-08-20]: Below are another couple of HTTP status codes, explanations and examples to aid in the more consistent development of true RESTful APIs:

301 "Moved Permanently": This is a good one when deprecating RESTful APIs. For example, say you are no longer going to support the resource (and all related individual CRUD and custom methods) /corporate_bonds and instead provide the following resource "/bonds/:type" where :type might include corporate, municipal and government. Whenever a valid request is made to the "/corporate_bond" resource I will return a 301 HTTP status to the consumer to inform them that they need to use the new API.
307 "Temporary Redirect": This is a nifty little status when you have temporarily suspended a resource, for example a "/users" resource. If a user is delinquent on their latest monthly installment, you want to start redirecting all calls to their profile, etc. to a temporary page that informs consumers of the RESTful site that this account is not current. There are probably many other good uses of this too. If you know of one, please leave a comment and educate me further.

Story-level BDD specing using rBehave

2007-08-09T12:51:00.000-05:00

For those in the BDD world that have been asleep at the wheel over the last month or two, Dan North has released rBehave which wraps around RSpec and allows developers to spec their code at the story level, which has been missed in RSpec (though I don't know what I would do without RSpec). Dan mentions that he is discussing integrating his work into RSpec at some point in the future, but in the meantime Dan's rBehave framework is currently at version 0.3.0 and is available for install as a Ruby Gem as usual: sudo gem install rbehave Dan gives an example of rBehave using a bank transfer story on his blog. Below I give another example of authenticating a user:

require ‘rubygems’
require ‘rbehave’
require ’spec’ # for "should" method

require ‘user’ # the actual application code

Story "authenticate user",
%(As a user of the system I wish to authenticate and access my account information.) do

  Scenario "correct password is supplied but user has been suspended" do
    Given "my user account is created", User.new do |user|
      @user = User.new
    end
    Given "my username is", "myusername" do |username|
      @user.username = username
    end
    Given "my password is", "mypassword" do |password|
      @user.password = password
    end
    Given "my user account has been suspended", true do |suspended|
      @user.suspended = suspended
    end
    When "I attempt to authenticate with", "mypassword" do |password|
      @authenticated = @user.authentiate?(password)
    end
    Then "my user account should respond with authentication status of", false do |status|
      @authenticated.should be(status)
    end
  end

  Scenario "correct password is supplied and user is active and verified" do
    Given "my user account is created", User.new
    Given "my username is", "myusername"
    Given "my password is", "mypassword"
    Given "my user account has been suspended", false
    Given "my user account has been activated", true do |activated|
      @user.activated = activated
    end
    When "I attempt to authenticate with", "mypassword"
    Then "my user account should respond with authentication status of", true
  end

  # A few other scenarios go here, like non-activated user (but not suspended), etc.
  # Left as exercise to the reader.
end

Please note, i do not need to restate the Given, When or Then blocks passed in after they are defined once!

Twitter4R v0.2.4 Released

2007-07-26T15:24:00.000-05:00

Version 0.2.4 of Twitter4R was released last night with the main change that makes the library much easier to use in Ruby on Rails applications. To install/update Twitter4R v0.2.4 just use the following command: sudo gem install twitter4r For more information on how to integrate Twitter4R with Rails see the following links:

My Blackle Energy Consumption Experiment

2007-07-24T11:14:00.000-05:00

I received a forward from a contact earlier today touting the energy savings of using Blackle instead of Google to search. My immediate thought was surely there would be no difference for an LCD monitor, or would there? I can, of course, see for CRT monitors energy savings are highly likely. The following is the energy saving content from the email forward I received:

We probably all use google several times a day - here's something to consider: When your screen is white, be it an empty word page, or the Google page, your computer consumes 74 watts, and when its black it consumes only 59 watts. An article about the energy saving that would be achieved if Google had a black screen, taking into account the huge number of page views, worked out at a saving of 750 mega watts/hour per year. In a response to this article Google created a black version of its search engine, called Blackle, with the exact same functions as the white version, but with a lower energy consumption, check it out: http://www.blackle.com

Since I work for a very energy conscious firm and I have been cycling or walking EVERYWHERE around town this summer (except I allow myself use of my car one afternoon a week maximum), I was curious and had to get to the bottom of it.

Hypothesis

There will not be any significant change in power usage (aka wattage) when viewing Google.com in my web browser versus viewing Blackle.com in the same web browser window. (I must confess I already knew how LCD monitors worked before conducting this experiment, but there are a surprising number of people who do not believe the backlight of an LCD monitor is on for the entire LCD regardless of what is being shown - ignorance is bliss for some).

Conditions

Dimensions of browser on screen must remain constant for both page views

Equipment

P3 Kill-A-Watt reader (x1)
17" Dell LCD monitor (x1)
[Indirectly used] Dell workstation (x1)
Cell phone (to time page view durations)

Preparation

Power off LCD monitor and unplug from power supply
Plug in power reader into power supply and plug LCD monitor plug into power reader socket
Select power reading (can choose from voltage, power and current "real-time" readings)
Record initial "power off" reading
Power on LCD monitor
Record initial "power on" reading

Method

Open web browser and maximize window
Surf to http://google.com
Start cell phone timer as soon as page is loaded
Monitor power reading for 30 seconds and note any changes to reading
In same web browser window surf to http://blackle.com
Start cell phone timers as soon as page is loaded
Monitor power reading for 30 seconds and note any changes to reading

Results

Initial "power off" reading: 0W (zero Watts) Initial "power on" reading: 20W (twenty Watts) after 3-4 seconds Google reading: started at 20W, no change for 30 seconds, ended at 20W. Blackle reading: started at 20W, no change for 30 seconds, ended at 20W.

Analysis

As we can see there was no significant change in the reading (to stated equipment error of plus or minus 0.2%). Therefore our hypothesis is shown to hold in this experiment using the equipment identified above and I conclude that Blackle has next to zero impact on energy consumption of LCD monitors. The good news is that my LCD monitor consumes much less than Blackle's claim of 50-75 Watts regardless of the search engine presentation I use!:)

Twitter4R v0.2.3 Released

2007-07-22T12:50:00.001-05:00

As promised earlier this week Twitter4R v0.2.3 made it out this weekend. Twitter4R v0.2.3 has just been uploaded to Rubyforge in Ruby Gem, tgz and zip formats. It may take up to several hours for the Rubyforge download mirrors to sync, so you may not be able to download it for a while. To upgrade/install using Ruby Gems: $ sudo gem install twitter4r I am in the process of updating the project website. So bear with me while I take care of this. Release Notes:

Fixed defect #31 such that passing string screen name as for user argument is handled correctly.
Fixed #30 typo: respond_to -> respond_to?
Added relevant exception handling for #message(:post, ...) case #32
Add ability to pass in Twitter::User object to Twitter::Client#user(...) #33
Added stats Rake task
Updated RDoc for Twitter::Client#user to warn against using it to get followers of authenticated user and updated ArgumentError raising logic as per #29.

Enjoy!

Twitter4R v0.2.2 Released

2007-07-18T17:21:00.000-05:00

As solo already noted in the comments for the Twitter4R v0.2.1 Released blog entry from yesterday, version 0.2.2 is already here:) I usually wait 4 hours before I blog about it to let the Rubyforge gem servers sync. Changes in the version 0.2.2 release include:

Fixed URI paths for user, messaging and friendship APIs (#25)
Added action checks for Twitter::Client methods: #user, #my, #message, #messages, #status, #timeline, #friend (#26)
Added 'source' configuration documentation.
Added missing attributes for Twitter::User (#28)

The most notable thing is that I add the protected and profile_image_url attributes to the Twitter::User model class, which were missing previously.

Twitter4R v0.2.1 Released

2007-07-17T10:30:00.000-05:00

To install/update Ruby gem use: sudo gem install twitter4r OR sudo gem update twitter4r Note: It may take some hours before the Rubyforge mirrors are in sync. The ‘source’ feature was added to Twitter v0.2.1 such that in the web interface Twitter users can see which application the messages were sent from. The default is ‘twitter4r’ with link ‘http://twitter4r.rubyforge.org’. You can change this, but you will also need to contact Twitter developers (not Twitter4R) to set it up for you on the server side. To set relevant client configuration in Twitter4R you can do the following:


require('rubygems')
gem('twitter4r', '>=0.2.1')
require('twitter')

Twitter::Client.configure do |conf|
  conf.source = 'mysourceidfromtwitter'
end

Remember you will need to contact the Twitter developers to setup a source id and url with them before the above code snippet will start working. Enjoy!

Twitter4R v0.2.0: Friendship API

2007-07-17T08:15:00.000-05:00

After a week I realized I had not yet written about the Friendship API in Twitter4R and it is one of the simpler the parts of the core library. Below shows some code to add and remove a friend:


require('rubygems')
gem('twitter4r', '>=0.2.0')
require('twitter')

client = Twitter::Client.new(:login => 'mylogin', :password => 'mypasswd')
# we can add a friend by using their unique integer id or user object.
screen_name = 'myfriend'
user = Twitter::User.find(screen_name)
id = user.id

# Like any of the following....
client.friend(:add, user)
client.friend(:add, id)

# We can also use any of the following APIs to do exactly the same thing...
client.friend(:remove, user)
client.friend(:remove, id)

See related Twitter4R RDoc.

Twitter4R v0.2.0: Extras API

2007-07-13T18:44:00.000-05:00

There are a couple of miscellaneous features in Twitter4R that I wasn't comfortable officially supporting like the featured users method in the Twitter REST API that is documented to do one thing and implemented to do another by the actual Twitter developers. Obviously I don't control what the Twitter developers do, so while I wanted to support this REST API in the library to achieve 100% Twitter REST API coverage I also wanted to emphasize that if the Twitter REST API implementation changes to work as documented then Twitter4R may not work well or at all for this method. To get an Array of Twitter::User objects that represent Twitter's current featured users we would write code like:


require('rubygems')
gem('twitter4r', '0.2.0')
require('twitter')

# Now we must include this extra file import or we will get a NoMethodError
require('twitter/extras')

client = Twitter::Client.new
users = client.featured(:users)

Also in the Twitter4R Extras API is a class helper method added to Twitter::Client called from_config which takes up to two arguments. Here is an example:


require('rubygems')
gem('twitter4r', '0.2.0')
require('twitter')

# I have a 'twitter.yml' file in subdir 'config' that I will load credential from.
# This feature of Twitter::Client is in the Extras API and you need to require
# twitter/console to access it.
require 'twitter/console'
config_file = File.join(File.dirname(__FILE__), 'config', 'twitter.yml')
twitter = Twitter::Client.from_config(config_file)

# By default the from_config method selects the credentials defined
# for the 'test' environment.  The twitter.yml file will look something like this:
#  test:
#    login: mylogin
#    password: mypasswd
# If, however, we wanted another environment's credentials to be used we pass that 
# in as the second argument to from_config like so...
twitter = Twitter::Client.from_config(config_file, 'dev')

Again here is a link to the Twitter4R v0.2.0 RDoc.

Twitter4R v0.2.0: User API

2007-07-13T18:33:00.000-05:00

The User API provides an easy Ruby API that maps onto a less consistent Twitter REST API, which is good news for Twitter4R users:) The following example code snippet shows how you would use the Twitter4R User API:


require('rubygems')
gem('twitter4r', '0.2.0')
require('twitter')

# I have a 'twitter.yml' file in subdir 'config' that I will load credential from.
# This feature of Twitter::Client is in the Extras API and you need to require
# twitter/console to access it.
require 'twitter/console'
config_file = File.join('config', 'twitter.yml')
twitter = Twitter::Client.from_config(config_file)

# Gets the Twitter::User object for user with screen name otherlogin.
user = twitter.user('otherlogin')
# Gets an Array of Twitter::User objects that represent the friends of user 
# with screen name otherlogin.
friends = twitter.user('otherlogin', :friends)

# Gets the Twitter::User object for the user we used to authenticate above
# using the Twitter::Client.from_config method in the Extras API above.
me = twitter.my(:info)

# Gets the Array of Twitter::User objects that represent the followers of 
# the user whose credentials were used to authenticate with the Twitter service.
myfollowers = twitter.my(:followers)
# The same can be done for friends, i.e.
myfriends = twitter.my(:friends)

# Alternative we can use the equivalent Model APIs like so....
myfollowers = me.followers
myfriends = me.friends

Hope you find this helpful. If you want to learn more about the User API you should consult the Twitter4R v0.2.0 RDoc.

Twitter4R v0.2.0: Messaging API

2007-07-10T09:18:00.000-05:00

Probably one of the more recent additions to the Twitter REST API has been that of direct messaging. In the Twitter web interface we were introduced to the supported construct of direct messaging only a couple of months ago. The Messaging API in Twitter4R provides access to these new supported features via two instance methods on the Twitter::Client class: #message and #messages. Below we can see an example of a common non-trivial usage of this API:


gem('twitter4r', '>=0.2.0')
require('twitter')

client = Twitter::Client.new(:login => 'mylogin', :password => 'mypassword')
received_messages = client.messages(:received)
# Now do whatever it is you wish to do with the Array returned and assigned to received_messages.
# It is an Array of Twitter::Message.  Twitter::Message has the following attributes:
# * sender (Twitter::User)
# * recipient (Twitter::User)
# * text (String)
# * created_at (Time)

new_message = client.message(:post, 'I am addicted to Twitter', 'myfriendslogin')
# Note: if we had a handle to our friend's Twitter::User object we could substitute that object
# for the screen name screen given (i.e. 'myfriendslogin' in the above example).  This would 
# look like:
#  new_message = client.message(:post, 'I am addicted to Twitter', user)

# Now we realized sending that message was a big mistake, so we try to delete is quickly:
deleted_message = client.message(:delete, new_message)

To find out more about the Messaging API of Twitter4R visit the v0.2.0 RDoc and select the examples/messaging.rb link from the top left frame. Also note that you can create new direct messages via the Model API of Twitter4R as well. Happy direct messaging on Twitter using Twitter4R.

Twitter4R v0.2.0: Model API

2007-07-10T08:53:00.000-05:00

For those that are familiar with the ActiveRecord API, the Model API of Twitter4R should feel fairly natural for you. Currently the raw Twitter REST API inherently exposes three types of models:

Status
User
Message

These model classes are available via the Twitter4R classes Twitter::Status, Twitter::User and Twitter::Message respectively. What you will notice on these classes is that they have the familiar find and create class methods that behave almost identically to ActiveRecord::Base subclasses (or models as we call them in Rails applications). There is one difference, due in part to the slightly different way "connections" to the Twitter4R "datasource" is handled. So below is a non-trivial example of using these ActiveRecord style methods:


gem('twitter4r', '>=0.2.0')
require('twitter')

client = Twitter::Client.new(:login => 'mylogin', :password => 'mypassword')
# I want to post a new status to my timeline, which can also be do by:
#  client.status(:post, 'My new status message')
status = Twitter::Status.create(:client => client, :text => 'My new status message')

user = Twitter::User.find('dictionary', client)
message = Twitter::Message.create(:client => client, :recipient => user, :text => 'canadaphile')

What we see here is very similar to ActiveRecord style models, but not quite. In the find method invoked on Twitter::User above we supplied a second argument, which is the client (the client context or "connection") object. We also make sure to pass in a :client key-value pair in each create method called above. Without it an ArgumentError would be raised by each create call. A few notes:

Twitter::User does not define a meaningful create method since Twitter doesn't allow the creation of new user account via their REST API
Twitter::User model has some class and instance helper methods, which are described more in the RDoc (see link below)

See the v0.2.0 RDoc for Twitter4R and select the examples/model.rb link from the top left frame.

Twitter4R v0.2.0: Status API

2007-07-08T21:35:00.000-05:00

The Status API deals with access to single status objects on the Twitter server. Since "updating" (in the traditional sense) a status in Twitter is non-existent (you update your timeline with a brand new status, but you cannot edit individual status text after created), Twitter4R provide a CRD-style interface (rather than true CRUD-style). For example, to post a new status to your timeline you might write code like the following:


gem('twitter4r', '>=0.2.0')
require('twitter')
client = Twitter::Client.new(:login => 'mylogin', :password => 'mypassword')
status = client.status(:post, 'Learning the Twitter4R Status API in 60 seconds.')

To retrieve the full status object (Twitter::Status instance - see Twitter4R Model API documentation) given a unique status ID you can code:


# Assume code in previous code snippet
status = client.status(:get, 140684282) # => should be announcement status of Twitter4R v0.2.0 release
# Now you can query the attributes of the status like...
puts "#{status.user.screen_name}: #{status.text} @ #{status.created_at}"

To delete the status we initially posted above (or any other status of yours that you have a handle to (a handle can be just the unique integer status ID or the status object itself) we can code:


# Again assume code from first code snippet above...
client.status(:delete, status) # here we can pass in the full object or just the id, it doesn't matter.  Same as in #status(:get, ...) case.

Upgrading from 0.1.x

To upgrade from Twitter4R v0.1.x you probably need to replace code like:


client.update('My status message text')

With code like the following:


client.status(:post, 'My status message text')

There were no APIs in Twitter4R v0.1.x for the single status :get and :delete use cases.

Twitter4R v0.2.0: Timeline API

2007-07-08T21:06:00.000-05:00

The Timeline API segment of Twitter4R provides access to the REST API that deals with public, friend and [public] user status timelines. For example, to get an Array of statuses representing your own timeline on Twitter all you would need to type was:


gem('twitter', '>=0.2.0')
require('twitter')
client = Twitter::Client.new(:login => 'mylogin', :password => 'mypassword')
timeline = client.timeline_for(:me)

If your timeline is public and you do not need to invoke any authenticated methods, you do not need to pass in login/password credentials to the Twitter::Client constructor, just Twitter::Client.new is all that is needed. Other than the :me timeline, the following are available:

:public - returns the public timeline on Twitter (last 20 statuses)
:friends - returns a timeline of all your friends' statuses
:friend - returns a timeline of one particular friend
:user - returns timeline of one particular user

Different options are available with each of the timelines described above. To read more consult the Twitter v0.2.0 official RDoc documentation.

Upgrading from 0.1.x

If you are upgrading from Twitter4R v0.1.x you will probably have code that looks similar to the following:


client = Twitter::Client.new(:login => 'mylogin', :password => 'mypassword')
timeline = client.public_timeline

All you will need to do is change the last (second) line above to:


timeline = client.timeline_for(:public)

Twitter4R v0.2.0: Configuration API

2007-07-08T14:48:00.000-05:00

The Configuration API allows applications using Twitter4R configure static connection options such as:

connection protocol (e.g. HTTP and SSL)
host name and port (good for testing or if the Twitter REST API moved to a different subdomain)
proxy settings (i.e. proxy host, port, username and password)
user agent identifier
X-Twitter-Client* HTTP headers for Twitter's internal usage analysis

This API intentionally tries to look like the Rails initializer/configuration API:


require('rubygems')
gem('twitter4r', '0.2.0')
require('twitter')

Twitter::Client.configure do |conf|
  # We can set Twitter4R to use :ssl or :http to connect to the Twitter API.
  # Defaults to :ssl
  conf.protocol = :ssl

  # We can set Twitter4R to use another host name (perhaps for internal
  # testing purposes).
  # Defaults to 'twitter.com'
  conf.host = 'twitter.com'

  # We can set Twitter4R to use another port (also for internal
  # testing purposes).
  # Defaults to 443
  conf.port = 443

  # We can set proxy information for Twitter4R
  # By default all following values are set to nil.
  conf.proxy_host = 'myproxy.host'
  conf.proxy_port = 8080
  conf.proxy_user = 'myuser'
  conf.proxy_pass = 'mypass'

  # We can also change the User-Agent and X-Twitter-Client* HTTP headers
  conf.user_agent = 'MyAppAgentName'
  conf.application_name = 'MyAppName'
  conf.application_version = 'v1.5.6'
  conf.application_url = 'http://myapp.url'
end

If using Twitter4R in a Rails application I would recommend adding the necessary Twitter4R configuration code in config/twitter.rb then including it in the config/environment.rb file to keep configurations separate.

Upgrading from 0.1.1

In Twitter4R v0.1.1 you could configure protocol, server host, server port and proxy settings using the following code:

Twitter::Client.conf(
  :ssl => true, 
  :port => 443, 
  :proxy_host => 'myproxy.host', 
  :proxy_port => 8080)

This API no longer exists, so if upgrading to Twitter4R 0.1.1 you will need to change the above to something like the following:

Twitter::Client.configure do |conf|
  conf.protocol = :ssl
  conf.port = 443
  conf.proxy_host = 'myproxy.host'
  conf.proxy_port = 8080
end

This change shouldn't be too painful for people I hope. Please refer to the Twitter4R v0.2.0 RDoc for more information.

Twitter4R 0.2.0 Release

2007-07-08T12:01:00.000-05:00

It's been almost two weeks, but Twitter4R had some major refactoring and updating between 0.1.1 and 0.2.0. So I hope you will think the wait was worth it. As usual you can grab the latest Ruby Gem (once all Rubyforge mirrors have synced) with: sudo gem install twitter4r. I am still in the process of pushing out the new Ruby Gem to Rubyforge and updating the website for the project, but the release is code complete. Some changes are quite significant in some areas. The basic idea of this release (0.2.0) was to cover the official published Twitter REST API 100% and remove some of the clutter the first two releases of Twitter4R introduced partly due to mirroring Twitter's inconsistent API. In the same fashion as both previous releases of Twitter4R, version 0.2.0 has 100% C0 code coverage via RSpec specifications. I separated out Twitter4R library into various API segments:

I will write a blog entry as an introduction to each segment over the next couple of days. In the meantime I will finish publishing the new website documentation and leave you with the RDoc for version 0.2.0. Updates:

New entry: Twitter4R v0.2.0: Configuration API
New entry: Twitter4R v0.2.0: Timeline API
New entry: Twitter4R v0.2.0: Status API
New entry: Twitter4R v0.2.0: Model API
New entry: Twitter4R v0.2.0: Messaging API
New entry: Twitter4R v0.2.0: User API
New entry: Twitter4R v0.2.0: Extras API
New entry: Twitter4R v0.2.0: Friendship API

Twitter API's unRESTfulness causing trouble

2007-07-05T11:07:00.000-05:00

I am now designing my sixth (6th) set of web services on my new project. The previous five (5) and this current venture were/are for the purpose of allowing partners and internal developers to integrate with a "platform" of some kind. These clients varied from Fortune 500 companies, prestigious privately held firms and most recently a VC-funded startup. The previous two web service APIs were based on the REST "architecture" (yes, I know that word gets used too much, but I use it now for lack of a better word at this moment). The previous REST API was in fact fully RESTful and my new software product design also has the aim of RESTfulness at all points. During my travels meandering through the windy trails of designing REST APIs, I have realized just how important being truly RESTful is to REST APIs. Yes, reread that sentence again. if it doesn't make sense I recommend reading RESTful Web Services, which should clarify this distinction for you. For me, the biggest plus of RESTful web services is that it promotes consistency at all levels. Moving on...where was I going? Ah, yes, Twitter... The routes.rb for twitter.com seem to be a little misleading. It seems a mix and match approach was taken to developing a "RESTful" API for twitter.com using Rails. The current RESTful Rails convention is the following (in a nutshell):

GET /resources/:id.:format => get information about resource with specified :id
POST /resources => create new resource using data given in request body
PUT /resources/:id.:format => update resource with specified :id using data given in request body
DELETE /resources/:id.:format => delete resource with specified :id and return resource in :format given

This is all great, however, in twitter.com it seems there was a previous ugly step-child of a "RESTful" design implemented before Rails 1.2 standardized on the above approach (just speculating here). The evidence of this exists in multiple places in the Twitter API's inconsistency, however, one case in particular creates a defect in my view (and I am being kind by not considering all RESTful violations as defects, since the Twitter developers need to cater to the weekend PHP and VB.NET hackers that might not care about the beauty of REST's simple yet consistent design and purpose - I will give the Twitter developers the benefit of the doubt for this article): In the users#show API there seems to be a conflict. For example, GET /users/show.json?id=mylogin will, in fact, return the user information for user with screen_name ‘show’ instead of ‘mylogin’, which is obviously not the intention of the above HTTP call, especially when /users/show is an advertised way to access user information in the official Twitter API docs. Yes, I know I can call GET /users/mylogin.json and be done with it, but the misleading defect still remains. Interestingly, GET /users/show?id=mylogin does yield the expected result assuming you have the Accept HTTP header set to the appropriate mime-type (e.g. 'text/x-json'). Screen names that cause potential conflicts (if using Rails defaults) are: create, destroy, update, new, show. All of which exist at twitter.com. My point here isn't to blast Twitter developers necessarily, simply to ask all developers (or if you must title yourself this way "architects") to consider the importance of creating consistent APIs, preferably based on open standards/recommendations like REST. And most importantly, if a REST API isn't truly "RESTful", don't say it is!:) You will get more respect for being honest from people that know what it really means. Related post:

Twitter4R v0.2.0 Released - the ultimate open source library that provides Ruby bindings for 100% of the officially documented Twitter REST API.

Marrying Autotest with RSpec on Gnome

2007-07-02T17:54:00.000-05:00

For those that don't know marrying Autotest with RSpec is a God send and I highly recommend it for pure Ruby as well as Rails application development. On the Mac OS X and KDE desktop environments there are builtin Autotest extensions that visually flag erroneous or successful RSpec runs with a standard desktop notification message. For Gnome users on Linux running Autotest with RSpec you may have seen a couple of ${HOME}/.autotest customizations out there that don't work with RSpec v1.x+. Below is my ${HOME}/.autotest that works with RSpec 1.0.5:


require('autotest/redgreen')
require('autotest/timestamp')

module Autotest::GnomeNotify

  # Time notification will be displayed before disappearing automatically
  EXPIRATION_IN_SECONDS = 3
  ERROR_STOCK_ICON = "gtk-dialog-error"
  SUCCESS_STOCK_ICON = "gtk-dialog-info"

  RE_RSPEC_SUMMARY = Regexp.new(/(\d+) examples, (\d+) failure/)

  class << self
    # Convenience method to send an error notification message
    #
    # [stock_icon]   Stock icon name of icon to display
    # [title]        Notification message title
    # [message]      Core message for the notification
    def notify(stock_icon, title, message)
      options = "-t #{EXPIRATION_IN_SECONDS * 2500} -i #{stock_icon}"
      system "notify-send #{options} '#{title}' '#{message}'"
    end

    def compose_message(at)
      specs, failures = 0, 0
      at.results.scan(RE_RSPEC_SUMMARY) do |s, f|
        specs = s.to_i
        failures = f.to_i
      end
      "#{specs} specs, #{failures} failures"
    end
  end

  Autotest.add_hook :red do |at|
    notify ERROR_STOCK_ICON, "Some specs failed.", compose_message(at)
  end

  Autotest.add_hook :green do |at|
    notify SUCCESS_STOCK_ICON, "All specs passed.  Have a beer!", compose_message(at)
  end
end

I assume there this will work with RSpec 1.x, but I have only tested with RSpec 1.0.5. Enjoy!

RSpec "should_yield"?

2007-07-01T16:51:00.000-05:00

While writing some specifications for the 0.2.0 version of Twitter4R (soon to be released - honestly), I realized there was no clean way to do #should_yield, so I thought I would offer my suggestion for the developers of RSpec to critique as appropriate. Assume we have a method signature that takes a block and should yield an object to the block given (if one is given). Something like the following:

class Client
  class << self
    def configure(&block)
      # implementation of Client.configure
    end
    # rest of class methods for Client class go here...
  end
  # rest of instance methods for Client class go here....
end

Where usage for the Client.configure method should look something like the following:

Client.configure do |conf|
  conf.proxy_host = 'myproxyhost'
  conf.proxy_port = 8080
  conf.proxy_user = 'me'
  conf.proxy_pass = 'mypass'
end

The object yielded to the block given is an instance of Config which has settable attributes: proxy_host, proxy_port, proxy_user, proxy_pass. The question is how do we write RSpec specifications for this? First we need to define what we want to specify for the expected behavior. My first attempt is to describe (in English) that:

Client.configure should
- accept a block as argument
- yield a Config instance to the given block

The way I did this in the Twitter4R (open source Ruby library for the Twitter REST API) project specifications was something like the following:

describe Client, ".configure" do
  before(:each) do
    @has_yielded = false
    @block = Proc.new do |conf|
      conf.is_a?(Config).should be_true
      @has_yielded = true
    end
  end

  it "should accept a block as argument" do
    lambda {
      Client.configure {}
    }.should_not raise_error
  end

  it "should yield a Config object to given block" do
    Client.configure(&block)
    @has_yielded.should be_true
  end

  after(:each) do
    @has_yielded, @block = nil
  end
end

What I would really like to do something like the following instead:

describe Client, ".configure" do
  it "should accept a block as argument" do
    lambda {
      Client.configure {}
    }.should_not raise_error
  end

  it "should yield a Config object to given block" do
    lambda {
      Client.configure(&spec_block(1))
    }.should yield_with(Config)
  end
end

I haven't extended the Proc class to this point, but if/when I do I will post it to this blog and perhaps also to the RSpec mailing list. If you have suggestions on how to improve the more manual way (the first RSpec code listing above), please do let me know. I have not yet played around with inheritable describe blocks yet, but I have hunch that would DRY up some of the code for if needed for many contexts. Alternatively if you have suggestions on how to improve the second RSpec code listing (directly above) to be more clear, please do let me know that too. I realize the developers of RSpec don't want to have too many #should_XXX methods, so I am open to suggestions (since I also agree with that notion). UPDATE: I created a new feature request and dialog with RSpec developers at: [#11949]

Good Riddance to Bad Rubbish

2007-06-27T07:30:00.000-05:00

While I usually do not express views about current affairs and political figures on this blog (except to highlight a point as applied to software), I couldn't let this beautiful day go by without a mention of the sheer delight I am experiencing being freed from the shackles and chains that was Tony Blair's premiership. My only regret is this has come ten years too late.
For those that do not know what our "fabulous" Tony has done domestically in UK, you should watch the British documentary Taking Liberties released earlier this summer. You can view the trailer below: If you still want to know more about the police state in UK created by Blair (all in the convenient name of "terrorism"), just ask Jean Charles de Menezes...oh wait, you can't. This Brazilian man was shot dead by Blair's police state at the Stockwell Tube station on the London underground. He was NOT affiliated to the London bombings in 2005 or any other terrorist plot or organization.

Twitter4R 0.1.1 Released

2007-06-25T12:36:00.000-05:00

Twitter4R, the open source Ruby project for the Twitter REST API, version 0.1.1 has been released this afternoon as a Ruby Gem. It may take a little time for the new Gem to appear on the Rubyforge gem servers, but once replicated to all mirrors you should be able to do the following:


require 'rubygems'
gem 'twitter4r', '0.1.1'
require 'twitter'

client = Twitter::Client.new(:login => 'mylogin', :password => 'mypassword')
timeline = client.public_timeline
timeline.each do |status|
  puts "#{status.user.screen_name}: #{status.message}"
end

This will get you started. Changes from 0.1.0 to 0.1.1 include:

Addition of proxy support
Addition of SSL support
Migration from RSpec 0.8.2 to 1.0.x specifications

For proxy support you can do the following:


require 'rubygems'
gem 'twitter4r', '0.1.1'
require 'twitter'

Twitter::Client.config(:proxy_host => 'myproxy.host', :proxy_port => 8080)
# continue using Twitter4R API as above.

Or if you have a proxy server that requires authentication, you can do the following:


Twitter::Client.config(:proxy_host => 'myproxy.host', :proxy_port => 8080, :proxy_user => 'myproxyuser', :proxy_pass => 'myproxypass')
# continue using Twitter4R API as above.

To force Twitter4R to use SSL, we would write the following configuration code before using the client API:


Twitter::Client.config(:ssl => true, :port => 443)

Enjoy!

Twitter4R Revamp

2007-06-25T09:14:00.002-05:00

While nothing has yet to be release for Twitter4R since version 0.1.0, it has not be standing still. A number of changes will be released within the next few business days. Firstly, I will be releasing Twitter4R 0.1.1 with just two public API changes (and one internal change):

[PUBLIC] SSL support
[PUBLIC] Proxy support
[INTERNAL] Migration from RSpec 0.8.2 to 1.0.x

Secondly, I will also be releasing Twitter4R 0.2.0 very soon after 0.1.1, which will contain a number of API changes. These changes are mostly documented in the comments of ticket #6 in Twitter4R's Retrospectiva project. I plan writing much more RDoc documentation for 0.2.0 and subsequent 0.2.x releases. Lack of documentation was a problem with the 0.1.0 release. The reasoning for the quick succession of releases of Twitter4R is so that 0.1.x versions can have the same public API, whereas 0.2.x signifies a change in API to allow us to cover more of the Twitter REST API without cluttering the Ruby API too much. A major reason for the API revamp was to hide some of the very unRESTful and asymmetric things the Twitter team have exposed in their public API. If any of the Twitter team are reading this, I recommend RESTful Web Services. I admit to dropping the ball on the next release of Twitter4R gladly, because it is true. I violated the first rule of agile development: Do not bite off more than you can chew! I apologize and will learn from it. Thanks for your patience.

Twitter4R API Changes

2007-06-16T19:15:00.000-05:00

Twitter4R 0.1.1, will have three new features:

Proxy support. Thanks to Kaiichi Matsunaga for donating the code.
SSL support. It will now be the default, but Twitter4R client can easily be configured to use HTTP instead
Major API refactoring such that Twitter4R feels more like Ruby and to obtain 100% programmable client coverage of the Twitter REST API. Refer to ticket #6 for more information on this

I'll be posting more on the changes and showing examples in later blog entries shortly after the release. Update: Refer to the following blog entries for updates to this:

The Java Dogma and Downfall

2007-06-15T09:40:00.000-05:00

Despite applying his talents towards a despicable cause, Joseph Goebbels (Hitler's propaganda minister from 1933) had amazing insights into public behavior and described his most famous observations:

If you tell a lie big enough and keep repeating it, people will eventually come to believe it. The lie can be maintained only for such time as the State can shield the people from the political, economic and/or military consequences of the lie. It thus becomes vitally important for the State to use all of its powers to repress dissent, for the truth is the mortal enemy of the lie, and thus by extension, the truth is the greatest enemy of the State.

Using Goebbels' wisdom Java™ successfully launched their propaganda campaign initially against C/C++ with the rhetoric that Java guards against programming errors unlike its predecessor. Yes, Java does lean very heavily toward a dictatorship style of governing. This is without doubt. But does this translate to guarding against developers' own stupidity? When moving from C++ and Perl to Java back in 1997 I appreciated this dictatorship-style of governing very much. It removed *many* cycles of thought from development and I became lazy. Earlier in this new millennium I moved to a more democratic language called Python, where I enjoyed this new found freedom. Soon after the Python doctrine embedded deep-rooted values in me, I discovered the [almost] Libertarian utopia that is Ruby (though not perfect, thus the [almost]). Where am I going with this? Excellent question. Where was I? Ah yes, Java™, propaganda, Goebbels, etc.... Now that Java™ has new mortal enemies - Ruby, Python and other non-static languages that provide significant productivity gains as compared with large bloated development in Java - the Java™ machine is using the same rhetoric. Is it working? In short, for the software Lemmings that don't think, it is working well. Due in large part to the Java Dogma. In truth I find the liberty offered by these non-static languages (yes, Python that still includes you, weird *strongly*-typed cousin) provide a much greater guard against programmer stupidity than the forced dictatorship style of Java™. This is because the developer and development team must take full ownership of the responsibility for their work and thus disciplined, smart and agile developers tend to flock to these languages, leaving the lazy developers only looking for a nice pay check for using the right buzzwords at the right time in the Java world. Java now faces a big problem: how to not let this lazy and uninspiring critical mass not weigh it's reputation down too much? This is an excellent question and one I fear Ruby will need to find an answer to soon, if PHP hackers with no real software development understanding keep flocking to Ruby through Ruby on Rails. The growing popularity of Ruby due to Rails will bring the current community (you and I) problems that we will need to start thinking about now if we are to survive the consequences. Or we will suffer a very similar fate to Java, which would be very sad indeed.

Are you a Cowboy Monkey?

2007-06-08T08:08:00.000-05:00

Everyone of us has probably met one. You may have worked for one as a subordinate. You may even be one. Though many people do not know what a Cowboy Monkey is. In a nutshell a Cowboy Monkey is that person on your team that jumps in to save the day with a hack. They are the ones that repeatedly put out the same fires instead of making sure they don't start them again. A Cowboy Monkey can't be bothered to take 5 minutes a day to commit code changes into a SCM and resync their working copy, but spends half the day in meetings not achieving anything. They are generally horrible dates because they spend all dinner talking about "quick and dirty" solutions. Generally a Cowboy Monkey:

does not believe in any type of SCM system.
does not think about the longer-term consequences of their actions only the immediate effects.
does not believe in unit, functional, integration, performance or system testing or specifications of any kind.
prefers to test in production rather than create a whole new staging environment.
has not heard of best practices.
has a tendency to build silo systems.
has a total disregard for the software architecture of a project.
usually has no real understanding of design patterns of any kind.
thinks PHP is so good that it can solve the war in Iraq.
gives smiles and nods but delivers very few of the promises they make and almost nothing on time.

If you nodded at least 5 times in agreement with any of the statements above, you might be a full-fledged Cowboy Monkey. Do not fear, you are amongst friends and we will help you overcome this disorder. If you are not one, how many Cowboy Monkey's have you worked with thus far? Please feel free to share your (anonymous please) stories of the Cowboy Monkey's you've met on the job in comments.

Higher Order Messaging

2007-06-04T11:37:00.000-05:00

In response to a devChix blog entry called RUBY: DRY up your Enumerations, I thought I would offer my alternative to the suggested notion of higher order messaging below:

# In reference to http://www.devchix.com/2007/05/25/ruby-dry-up-your-enumerations/
# Examples by Susan Potter [http://susanpotter.net]

# Framework code...

class ProxyBase
  attr_accessor :collection
  
  def initialize(collection)
    @collection = collection
  end
end

class AnyProxy < ProxyBase
  def method_missing(sym, *args)
    @collection.each do |item|
      return true if item.send(sym, *args)
    end
  end
end

class AllProxy < ProxyBase
  def method_missing(sym, *args)
    @collection.each do |item|
      return false unless item.send(sym, *args)
    end
  end
end

module LinkingVerbMixin
  def self.included(base)
    base.extend(ClassMethods)
  end
  
  module ClassMethods
    def any(collection)
      AnyProxy.new(collection)
    end
    
    def all(collection)
      AllProxy.new(collection)
    end
  end
end

class Are
  include LinkingVerbMixin
end

class Do
  include LinkingVerbMixin
end

# Application code...

class Stooge
  attr_accessor :name, :bald
  
  def initialize(name, bald)
    @name = name
    @bald = bald
  end
  
  def bald?
    @bald
  end
end

mo = Stooge.new("Mo", true)
larry = Stooge.new("Larry", false)
curly = Stooge.new("Curly", false)
stooges = [mo, larry, curly]

Are.all(stooges).bald?
Are.any(stooges).bald?

# This can be extended further to do stuff like Do.all(stooges).have(:name, :eql?, "Mo"), etc.

Or see the pastie.

RSpec 1.0.0 Finally Here!

2007-05-19T15:32:00.000-05:00

Thanks to the RSpec team for finally delivering RSpec, which I have been excited about for quite some time. Having about 5 projects (2 pure Ruby, 3 Rails) using RSpec 0.8.2 and 2 other projects still on (0.7.5) I was a little worried about migrating to the new RSpec 1.0.0 compliant API, which is a fair bit different than some of what was allowed in 0.8.2 and prior. However, the RSpec developers in all their wisdom foresaw this problem and wrote spec_translate where you simply give it the from directory and to directory in that order on the command line. Now it doesn't do everything, so don't expect too much, but it will convert all the context, setup, specify and teardown blocks to describe, before, it and after blocks respectively with very little fanfare or effort required from us. The gotchas for those of us that ignored the Test::Unit Cheat Sheet on the website for recommended RSpec usage (I am going to own up to it, yes):

.should.eql expected_value becomes .should eql(expected_value)
.should.not... becomes .should not...
.should.be expected_result becomes .should be(expected_result)

Let me know if you find different varieties of the above API differences.

Are you a Pig or a Chicken?

2007-05-17T20:06:00.000-05:00

Grey's Anatomy fans amongst us may remember an episode last season (season two) where a patient's boyfriend introduced us to the idea of whether you are a Pig or a Chicken in a relationship. The idea is that when you have a nice greasy breakfast of bacon and eggs, you have the Pig who is completely committed to delivering the breakfast and the Chicken who is only involved in contributing to the breakfast. So in the Grey's Anatomy episode the boyfriend was trying to be a Pig in his relationship and be fully committed to his heavily pregnant girlfriend. For me this wasn't a new concept, having read about Scrum as a software project management practice or adaptive process (if we are really forced to use the word "process") over the last 3 years. Now in Scrum the idea is the same, except applied to the software project not a relationship between two people. The Pigs are generally the people who are 100% committed to the project/product (e.g. Developers, DBAs, BAs, QAs, etc.) and the Chickens are the people that have a stake in the project but only contribute to the project. They are not 100% committed (e.g. business stakeholders, management, etc.) just involved in the project. Are you a Pig or a Chicken?

Ruby Equality

2007-05-14T12:48:00.000-05:00

Since two people asked me in the space of 5 days about this, I thought I'd post a quick demystification of Ruby's "equality" facilities. Namely equal?, ==, eql? and ===.

equal? - always (or should be) purely identity equality, and *should never* be overridden by subclasses
== - by default is object identity equality, but *can* be overridden by subclasses
eql? - is object value equality and *should* be overridden by subclasses
=== - by default same as #==. It is overridden by subclasses that need a "case" equality defined to make sense when used in the case statements in a when construct.

ActiveRecord::Base#to_xml woes

2007-05-08T23:10:00.000-05:00

For us Railies that use namespaced models inside Ruby modules and are relying on ActiveRecord::Base#to_xml, do not fret any longer. To reproduce the problem you just need to do something like:


users = Legacy::User.find(:all) #=> []
users.to_xml #=> <?xml version=\"1.0\" encoding=\"UTF-8\"?><legacy/users></legacy/users>

The XML produced by #to_xml is NOT valid XML for this case. It's been a small thorn in my side for a week or so now and I've been living with it by adding the following to a file that is eventually/implicitly required by environment.rb:

# Patch for Rails #8305 that I submitted on 2007-05-09
# We'll need to keep an eye out upon next Rails Gem upgrade whether or not this 
# fix has been applied to remove this or not.
require 'active_record'
require 'active_support'

class ActiveRecord::Base
  alias :old_to_xml :to_xml
  def to_xml(options = {}, &block)
    options[:root] ||= self.class.name.tableize.singularize.gsub(/\//, ':') # for models that are namespaced in Ruby module
    old_to_xml(options, &block)
  end
end

module ContainerPatchMixin
  def self.included(base)
    base.class_eval %{
      alias :old_to_xml :to_xml
    }
    base.send(:include, InstanceMethods)
  end
  
  module InstanceMethods
    def to_xml(options = {})
      contained_class = self.first.class.name
      options[:root] ||= contained_class.tableize.gsub(/\//, ':') # for models that are namespaced in Ruby module
      options[:children] ||= contained_class.tableize.singularize.gsub(/\//, ':') # for models that are namespaced in Ruby module
      old_to_xml(options)
    end
  end
end

class Array
  include ContainerPatchMixin
end

Now what is produced is something more along the lines of:


<?xml version=\"1.0\" encoding=\"UTF-8\"?><legacy:users></legacy:users>

Which is valid XML. Today I finally reported this edge case defect on dev.rubyonrails.org (#8305) as I hadn't seen anyone else report it yet. Thankfully this is Ruby and this is all possible while the Rails Core team determine if my not so elegant fix is worthy to be in the next Rails version or not.

Twitter4R 0.1.0 Released

2007-05-06T23:55:00.000-05:00

This weekend I had a couple of hours to work on Twitter4R (finally). Today (Sunday) saw the release of Twitter4R 0.1.0 on Rubyforge as a Gem. It may take the Rubyforge gem mirrors a few hours to copy it over, but by Monday morning (US time) you should be able to do the following:


$ sudo gem install twitter4r
$ irb
irb(main):001:0> require 'twitter'
=> true
irb(main):002:0> client = Twitter::Client.new :login =>
'screen_name', :password => 'password'
=> #
irb(main):003:0> statuses = client.public_timeline
=> [#, ...]
irb(main):004:0> statuses.each do |status|
irb(main):005:1*   puts status.text, status.user.screen_name
irb(main):006:1> end
the puts output would go here....
irb(main):007:0> client.update 'Pure Ruby API for Twitter:
http://twitter4r.rubyforge.org'
=> nil

It supports utf8 strings so this should not be a worry when querying the public timeline! See Twitter4R Rubyforge Website for more details or the Twitter4R Project Blog. Enjoy!

Favorite Quote of the Week

2007-05-03T11:29:00.000-05:00

A friend and old co-worker, who wants to be referred to as Bo "Fucking" Conroy, just sent me this on IM:

yeah, php seems like something targeted at 16-22 year old porn site developers.

Thanks Bo! Only you can verbalize these delicate things so precisely.

Rails Application Scaling Tips

2007-05-03T09:34:00.005-05:00

This will be a howto on quickly enhancing performance, throughput or uptime of a Rails application:

Use memcached for sessions
Move static assets to separate lighttpd or httpd web server optimized for just static content
Use more efficient templating engine such as erubis
Use memcached for other high-frequency read AR models
Configure Rails to cache pages, actions and fragments where relevant
Use BackgrounDRB to delegate longer running tasks such as sending emails, etc. Note: This is out of date right now, I might recommend Cloud Crowd or other plugins at this point for background jobs or even using message oriented middleware (MOMs) like AMQP services like RabbitMQ.
If your application requires a large number of concurrent large file uploads (say you are building the next YouTube) you should consider utilizing Merb. Note: this is also out of date since I wrote it in May 2007.

At this point you just need to focus on the standard scaling concerns:

database server tweaking (including database caching)
removing unnecessary file system I/O
removing unnecessary or consolidating network I/O
Removing unnecessary computation or caching computational results where necessary
etc.

Update (2010-08-15)

The following tools/plugins can be used to optimize the how many and how much data is passed by the SQL queries generated by ActiveRecord:

Bullet - helps you optimize your SQL queries by notifying you when not eager loading or using counter caches.
Scrooge - SQL query optimizer which can reduce the amount of data getting sent from your database to your Rails application.
Rack-bug - reports stats about each request in development mode so you can see how many queries are being executed on each request (and a number of other useful request stats).

For identifying memory bloat or leaks in the applications the following might be useful (when we have a need for such tools):

Memorylogic - logging memory usage.
Oink - for easily finding actions that significantly increase VM heap size.
BleakHouse - used this before for finding memory leaks in a Rails 1.2.3 app a couple of years back. I believe it is working against Rails 2.3.4 apps now too.

Also nesting includes on eager loading in ActiveRecord used to have memory leak implications. I wrote about my experience in 2007 here: Twas the night before launch.

The memory leak from the Twas the night before launch blog post may have been fixed in subsequent releases of ActiveRecord since July 2007 when I encountered it, in the case of nesting includes for eager loading in all one query, but the morale of the story is never assume that just because you are eager loading all in one query there aren't going to be other implications.

Registering Rails File Extensions in GEdit

2007-04-30T20:59:00.000-05:00

I just wanted to slap a quick howto for a couple of my professional and open source associates to get going with GEdit quickly while developing Ruby on Rails applications. GEdit uses Gnome's VFS for mime-type determination. To register new file mime-types for Gnome's VF do the following:

Create XML file with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
<mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
  <mime-type type="application/x-ruby">
    <alias type="application/ruby" />
    <comment xml:lang="en">Ruby Rake Tasks</comment>
    <glob pattern="*.rake"/>
  </mime-type>
  <mime-type type="application/x-ruby">
    <comment xml:lang="en">Rakefile</comment>
    <glob pattern="Rakefile"/>
  </mime-type>
  <mime-type type="text/html">
    <comment xml:lang="en">eRB HTML Markup</comment>
    <glob pattern="*.rhtml"/>
  </mime-type>
  <mime-type type="application/x-javascript">
    <comment xml:lang="en">eRB JavaScript</comment>
    <glob pattern="*.rjs"/>
  </mime-type>
  <mime-type type="text/xml">
    <comment xml:lang="en">eRB XML</comment>
    <glob pattern="*.rxml"/>
  </mime-type>
  <mime-type type="text/x-yaml">
    <comment xml:lang="en">YAML</comment>
    <glob pattern="*.yml"/>
  </mime-type>
</mime-info>

Copy file to /usr/share/mime/packages/rails.xml
Execute sudo update-mime-database /usr/share/mime
Enjoy!

Is Linux Now Cool?

2007-04-20T11:21:00.000-05:00

Injecting Understanding

2007-04-19T15:42:00.000-05:00

Over the past 4 weeks numerous people (Java heads trying out Ruby for the first time, or even coding neophytes that are jumping on board the Ruby train) have asked me about the .inject method. So I will try to demystify a method that (to me) epitomizes Ruby, yet baffles many that first encounter it. First, let's start with the basic concept. Here is some code you might find around in Ruby examples, etc., to demonstrate the purpose of inject:

(1..10).inject(0) { |cumulative_sum, item| cumulative_sum + item } #=> 55

Let's dissect:

(1..10) is an inclusive Range (if you don't know what a Range is in Ruby...better read up here, as it is out of scope for this article) from 1 to 10 (inclusive).
Then because Range is Enumerable (again if you don't know what Enumerable in Ruby means...read up about it here, again out of scope of this posting) it has inject available, so it bears remembering that it isn't just Arrays inject will work on.
We initially pass in 0 (yes, that's zero) as the argument to inject, which tells inject to start the "balance" at 0 (yes, zero again). Then it iterates through all the elements in the Range from 1 to 10 (inclusively remember because we used two dots not three) passing in the cumulative sum (or balance) and the current item into the block you define. The block we defined above is an expression that evaluates to an integer in this case.
The result of this expression is now set to the cumulative_sum (or "balance") carried forward into the next iteration. So as we should have all learned in school Mathematics when getting to summation series', the cumulative sum of 1 through 10 is 55 (this is the result we get from inject from the above code - and not by accident either - this is exactly what inject is all about).

You might also remember (or perhaps it is just sad Math geeks like myself that remember this stuff) that the sum of any set of consecutive numbers (starting from 1) is defined the by formula: (N*(N + 1))/2, where N is the largest of all consecutive numbers in the set. With this knowledge we can sanity check (for our own understanding) whether it does what you should expect it to do or if it was just coincidence above:


def sum(n)
 n*(n+1)/2
end

(1..9).inject(0) { |cumulative_sum, item| cumulative_sum + item } #=> 45
sum(9) #=> 45

(1..100).inject(0) { |cumulative_sum, item| cumulative_sum + item } #=> 5050
sum(100) #=> 100

(1..7890).inject(0) { |cumulative_sum, item| cumulative_sum + item } #=> 31129995
sum(7890) #=> 31129995

See told you it wasn't just coincidence - or what everything above a coincidence? Hmmmm..let's not go there right now!:). I'll bet you could very easily figure out the more general formula for any sum of consecutive positive integers not necessarily starting at 1 (yes, that is one). At least that is what my Mathematics teacher would ask us to do for homework (really not very much homework to be honest - I had a nice teacher)! I'll not make you prove it by induction today though;) Maybe tomorrow... Now you can do more interesting things instead of just summing consecutive positive integers. Don't just move on to geometric progressions either - the world is your oyster...like finding numerical solutions using an iterative method of your choice using this streamlined apporach!

The Beautiful Code Debate

2007-03-28T20:53:00.000-05:00

On my reading travels I came across Machine Beauty: Elegance and the Heart of Computing. For those still debating whether or not the pursuit of beauty is beneficial it is a must read. Another resource on this topic is David Heinemeier Hanson's talk on Rails: The Pursuit of Beauty.

Twittering Around in Ruby

2007-03-19T18:37:00.000-05:00

I just spent 2 minutes hacking around with twitter.com and thought I would share since a friend of mine wanted to use twitter to automate a few tasks for his websites in a similar vain to Urban Dictionary's use of twitter:


require 'net/http'
require 'uri'
require 'json'

HOST='twitter.com'
PORT=80
URI='/statuses/friends_timeline.json'

Net::HTTP.start(HOST, PORT) do |http|
  request = Net::HTTP::Get.new(URI)
  request.basic_auth('login', 'password')
  response = http.request(request)

  #Do something with the response.
  puts "Code: #{response.code}" 
  puts "Message: #{response.message}"
  puts "Body:\n #{response.body}"
  # to parse the JSON do something like: 
  #hash = JSON.parse(response.body)
  #and then do something with it.
end

Prep work before you run this Ruby code is:

$ sudo gem install json

WARNING: The fastest Rails application on the web, Twitter aint! I tried running against SSL, but got timed out too many times to test. Though if you aren't having problems you can try the SSL version of the code:


require 'net/https'
require 'uri'
require 'json'

HOST='twitter.com'
PORT=443
URI='/statuses/friends_timeline.json'

http = Net::HTTP.new(HOST, PORT)
http.use_ssl = true
http.start do |http|
  request = Net::HTTP::Get.new(URI)
  request.basic_auth('login', 'password')
  response = http.request(request)

  #Do something with the response.
  puts "Code: #{response.code}" 
  puts "Message: #{response.message}"
  puts "Body:\n #{response.body}"
  # to parse the JSON do something like: 
  #hash = JSON.parse(response.body)
  #and then do something with it.
end

You will get a warning when running it like the following:

warning: peer certificate won't be verified in this SSL session

Though if you refer to the Net::HTTP rdoc you will find how to work with this if you wish. I should also mention that you can edit the URI to the following values to get at different types of data:

/statuses/public_timeline.json - JSON representation of public timeline on twitter
/statuses/friends.json - JSON representation of friends and their current messages on twitter
/statuses/followers.json - JSON representation of your followers and current messages on twitter

Two things to note with the above:

By substituting .json with .xml twitter will send back XML format.
/statuses/update.json can be used in a HTTP POST request sent with a status=message+url+encoded name-value pair to update your own status. To accomplish this you will just need to swap out the request = Net::HTTP::GET.new line of code with something like the following: request = Net::HTTP::Post.new(URI, {'status' => 'my new message'})

Hope this helps!

Was Buddha Agile?

2007-03-19T10:14:00.000-05:00

I just found two interesting Buddha quotes this morning:

Ennui has made more gamblers than avarice, more drunkards than thirst, and perhaps as many suicides as despair.

The above quote demonstrates a mindset to dissuade ourselves from getting stuck in a rut through boredom. I've seen the effects of boredom on traditional software projects to know the above prophecy to be very true in this application.

I never see what has been done; I only see what remains to be done.

This next quote captures (to me) the essence of agile development practices: focusing on what needs to be accomplished rather than composing lengthy status reports for clients or managers to look like you are doing "stuff". I've worked with far too many "hot air" consultants from large consulting firms yelling the latest (at the time) buzz acronyms such as SOA, ESB, EAI, etc to know that these types of individuals do not accomplish anything in the real world.

Getting Local with Illinilist.NET

2007-02-19T16:59:00.000-06:00

Living in a small, sleepy college town I have realized how few online resources there really are that are (a) usable; (b) not regularly spammed; (c) not just for advertising purposes; and (d) not a poor translation of an offline concept, such as classifieds in particular. Yes, I know there is a Champaign-Urbana Craigslist, but let's be honest, the smaller Craigslist sub-sites are spammed without regard AND does anyone really think the website is all that nice to use? You would think after being in production for so many years and having so many users they would have come up with an "account" where you can manage all your listings well before last year! Not to mention the fact that Craigslist is a one-size-fits-all, cookie cutter approach to "going local". Each sub-site is exactly the same except the content. Is Craigslist really local? I think not! As for locally run websites: I have found a couple, but they look like they were all coded in the 1990s with no ability to do anything functional (i.e. it is a static site that needs to be manually updated) or the website is just an excuse to put up big advertisements with inconsequential substance. Now I am no web design fashionista, so my comments on the local websites written in the 1990's might be a little below the belt, but there is a lot that web developers should have realized since the 1990's in terms of usability on the web, not just how to make it look prettier. After being frustrated for far too long with the status quo, I finally decided to put a website together relatively quickly to begin to create an online venue for classifieds (but hopefully translated to the web in a meaningful way and for the target audience). Illinilist.NET was officially launched last week, but haven't had the time to really promote yet. So if you are a Champaign-Urbana resident looking for such an online local resource, check it out and let me know what you think about it. Illinilist.NET currently only supports browsing, searching (by tags) and posting Events, Jobs and Rentals for the local area. However, I am planning on beefing up the search capabilities, especially for Rental postings AND adding Real Estate, Resume and Personal listings within the next month. See the Illinilist.NET Blog to keep informed of the latest developments.

SCM: The Next Generation

2007-01-25T21:58:00.000-06:00

I am sure most developers (whether Java, Ruby, Python, or other) have used Subversion, Perforce or CVS to manage and control their projects' source, but have you used darcs or git? I am sure most of you would have heard of git - it hosts the Linux kernel code currently. A few less have probably encountered darcs before on an obscure European open source project you were interested in once or twice. Prior to this evening, the most notable point of darcs, to me, is that it is written in Haskell - a language I plan on reading more about because I have heard so many good things about it. Both git and darcs are examples of distributed version control systems of which there has been an explosion of products (both commercial and open source) since GNU's Arch. Tonight I had the fortune of experiencing darcs for the first time. Previously I had used darcs get URL to download code before, but never actually utilized any of the features before. Tonight I submitted a patch to a Ruby open source project (RForum) and found the experience very pleasurable as a developer. I have read that darcs is a user friendly version of git minus some powerful options. Perhaps there are git or darcs gurus reading that can shed more light on these two less mainstream SCM systems? If so, I would love to hear your thoughts on them especially if you can compare and contrast with any non-trivial Subversion, Perforce or CVS experiences. Background: I currently use Subversion, however, I usually work in smaller teams, so concurrent work is not that much of an issue at present, but things will be changing. The largest team I worked in on a Subversion project was 6 active developers (9 for the one Perforce project). Of course, I have worked on a project with over 80 developers, but that was using ClearCase and there were so many problems with that project that I can't determine what was caused by ClearCase or the rest of the project's [many] issues.

Ruby Idioms, Part 7

2007-01-24T18:48:00.000-06:00

This idiom I have seen a little more in Rails than I have seen in Ruby, but I am putting it in this Ruby Idioms series anyway. First off, most of you will know by know that you can "transparently" provide a method in Ruby a Hash. What do I mean by "transparently"? Well have a look at the code example below:


user = User.find(:first, :include => :preferences)

The last argument is actually a Hash even though you do not see the curly braces at all. In my opinion it makes for more readable code. So let us look behind the scenes at what the User.find method might do if it wasn't a magic method from ActiveRecord:


class User
  # ... some stuff here
  class << self
    def find(type, params = {})
      options = @@DEFAULTS.merge(params)
      case type
      when :first
        options[:limit] = 1
      when :all
        # do stuff with options and query database
        # bla bla bla
      end
      # rest of implementation
    end
  end
  # ... some stuff here
end

This is quite useful, but one of the idioms in the same vain that I really appreciate is the following usage:


  validates_length_of :name, :title, :company, :in => 2..128

This provides another example of supplying a transparent Hash arguments at the end of a list of arguments, the length of which is unknown. For example we may decide that the code needs to actually be more like:


  validates_length_of :name, :company, :in => 2..64
  validates_length_of :title, :in => 2..128, :allow_nil => true # to cater for those ridiculously long job titles out there or those that do not have a job title at all

Now you will notice that the validates_length_of method accepts a list of variable size for the list of attribute symbols to check and then at the end an option hash to specify length checking specifics. How does the method know there is an option Hash at the end to use for these things and/or how long the list of attributes is? The secret is in the sauce. If we were to look in the source code (though I am not, because I have seen it so many times before), we would see something like:


def validates_length_of(*attrs)
  options = @@DEFAULT_OPTIONS.merge(attrs.pop.symbolize_keys) if attrs.last.is_a?(Hash)
  # then continue with rest of implementation.
end

All it is doing here is in the signature, specifying that there is a list of unknown length to be supplied to the method. Then it pops the last item off that list (Array) if and only if the last element of the list is a Hash for use as the options that the implementation will use. Until the next time enjoy Ruby's "transparent" Hash idioms.

Ruby Idioms, Part 6

2007-01-23T22:28:00.000-06:00

Tonight I remembered couple more idioms in Ruby and felt compelled to share. In Javafied Ruby code (below) we often see something like the following:

if something
  for role in user.roles
    return true if @@roles.include? role
  end
end

Now when I first started writing Ruby code almost 3 years ago, I thought looping through a collection like above was the nicest way. I quickly found Ruby like the following snippet in code I inherited:

user.roles.each do |role|
  return true if @@roles.include? role
end

At first it took me some time to understand it (remember I was just a poor Java developer with a little Python experience), and I wondered why people would write loops this way. At the time it made no sense. Using this form of looping in Ruby does somewhat violate my "reads more like natural language" coding style preferences, but for some reason this form of looping is now the only one I use in Ruby code. Despite this violation, it feels much more natural. If you are unsure about this yourself, I would encourage you to look into the power of blocks in Ruby. Once you fully appreciate blocks, I think you might be persuaded, (though different opinions make this world more colorful and less bland). The second Ruby idiom is less odd and does enhance code readability more like natural language (again in my opinion):

user.roles.each do |role|
  return true if @@roles.include? role
end if something

Usually when people are having a conversation in English (at least native English speakers), we tend to say things like: We need to verify at least one role the user belongs to is in a pre-defined list of roles if [something]. Non-techies tend to say "if something" at the end of sentences, not at the very beginning (at least this is my experience). So for this reason I think this idiom enhances readability of the code to be more like natural English. For those that start every sentence that contains a conditional with the conditional clause, you have probably been a developer for far too long - go be a beach bum to save yourself before it is too late.

Ruby, Javascript, Erlang blog: Snakes, Gems & Coffee

DRAFT: Why proponents of marriage equality should love graph databases, Part 2: A Reply to 'The Database Engineering Perspective'

New Concepts in Graph Databases

How can graph databases overcome relational database shortcomings?

Gay marriage: the graph database solution

Why proponents of marriage equality should love graph databases, Part 1: A Reply to 'The Database Engineering Perspective'

What is wrong with relational databases?

What is coming in Part 2?

What is coming in Part 3?

How agile practices improve code review

Values

Principles/Tenets

Practices

So Long, Farewell, Auf Wiedersehen, Goodbye...to Ruby & Rails (sort of)

Thanks mates!

But...

Cloud Computing, Part 2: Law, Privacy & Maturity

Cloud Computing, Part 1: Delivery Models

Alan Turing Can Now Rest In Peace

Faker, Ruby on Rails Snippet & Generator

Design Patterns Book Club

Google Wave is not the answer, XMPP is!

Related Links

Erlang doesn't support currying? News to me!

Twitter4R 0.3.1 and the future

The Scala vs Erlang Debate, Part 2: The Geek Off

Scala: Technical Overview

Erlang: Technical Overview

Scala: Concurrency

Erlang: Concurrency

Scala vs. Erlang: Performance

Conclusion

Limitations of this series:

Resources:

Corrections (added February 2010):

The Scala vs Erlang Debate, Part 1: The Managers Overview

The Legacy Enterprise

Startups

Resources

Programming Erlang: Software for a Concurrent World by Joe Armstrong

Real World Haskell

CUBarCamp: Champaign-Urbana's Tech Meetup Update

Want to participate?

YUI MenuBar: A Facebook-like Application Bar

CUBarCamp: Champaign-Urbana's Tech Meetup 4/22 (tentatively)

Coworking Space in Champaign Urbana (Illinois)

GEdit Extensions Project

Free Coffee for Voting

Book Shelf Clearing

Tips to Enterpreneurs Hiring Ruby on Rails Consulting Firms

DataMapper: Flexibility of mapping model attributes to table columns

DataMapper does have migrations

Next Generation Syndication and Publishing

NEVER use or trust Paypal

LA Times Travel uses Twitter4R

Serendipity: An original Pradipta's Rolodex member

Twitter4R Development Releases

Rails rocks, X sucks: How Provincial

Why Merb is delicious

Comedy of Errors

Coding with Explicit Intentions

Waking Up From Hibernation

Preventing Information Leaks, Part 3

Seek Unguarded Hidden Files

Preventing Information Leaks, Part 2

HTTP Header Leakage

DataMapper vs. ActiveRecord

Preventing Information Leaks, Part 1

Identifying the framework or "stack"

Stupid Easy: 7 Deadly [Rails] Sins, Part 3

Stupid Easy Ruby

Ruby Language Pointers

Stupid Easy: 7 Deadly [Rails] Sins, Part 2

4. Stupid Easy Views

5. Stupid Easy Configuration (and Routes)

6. Stupid Easy Library Code

Stupid Easy: 7 Deadly [Rails] Sins, Part 1

1. Stupid Easy Schemas

2. Stupid Easy Models

3. Stupid Easy Controller Methods