Steve's Blog

New blog

2008-05-20T03:03:00.000-07:00

I'm ditching blogspot and using my own install of Radiant at iannopollo.com. Check it out

Mail Fetcher - Fetch Email from Ruby and Rails

2008-02-17T14:24:00.000-08:00

I recently had to write something to fetch emails from a GMail account and process the emails to check if the email was a bounce (bad email address). If so, the person in the system associated with the bounced email would be deleted from the database. I did some searching for such thing and found this:

How to receive emails with Rails

About half way down the page you see an example for polling emails from a POP3 server (edited for space):


require 'net/pop'
require File.dirname(__FILE__) + '/../config/environment'

logger = RAILS_DEFAULT_LOGGER

logger.info "Running Mail Importer..." 
Net::POP3.start("mail.server.net", nil, "username", "password") do |pop|
  if pop.mails.empty?
    logger.info "NO MAIL" 
  else
    pop.mails.each do |email|

      begin
        logger.info "receiving mail..." 
        Mailman.receive(TMail::Mail.parse(email.pop))
        email.delete
      rescue Exception => e
        logger.error e.message
      end

    end
  end
end
logger.info "Finished Mail Importer."

So try this with a GMail server and you get no connection. Why? Because Google uses SSL when logging in via POP3. So the I looked around and found someone using SSL to login via POP3. All they did was add this line right before trying to start the connection: Net::POP3.enable_ssl(OpenSSL::SSL::VERIFY_NONE)

Simple enough. I went ahead and tried that, and it blows up with the following error: undefined method `enable_ssl' for Net::POP3:Class. Now trying to find out why this didn't work was annoying, but I finally found out that in the current version of Ruby (1.8.6), Net::POP3 doesn't understand enable_ssl. The edge version, however, does. So, I got the net/pop.rb file and required it instead of the 1.8.6 version and voila! It works! So, instead of just posting this and letting everyone know that you have to do this to make SSL POP3 access work, I made a plugin:

Mail Fetcher

Now the whole world may enjoy SSL POP3 access to their hearts' content. And, it also works with IMAP. Checkout the README and mail_fetcher.rb to get all the details. Enjoy!

Talking to Gmail with Ruby and Rails

2008-02-11T13:12:00.000-08:00

Recently I came across a dilemma in trying to send and receive emails from a Rails app: Either install a full-blown postfix server on a Linux box (by full-blown, I mean account handling, POP/IMAP interface, anti-spam, anti-virus, etc etc), or use Google to handle all the emailing. So, me not being a Linux/sys-admin junkie, I strongly opted for the Google route (and after some persuasion the customer, thankfully, agreed :-). So I went about to find the best practices for working with Google Apps (http://www.google.com/a which is where you sign up). I came across this very helpful post.

You just install the mentioned plugin, create the appropriate mailer.yml file and you're good to go. ActionMailer will login as the specified account and send emails through that account. Genius!

On a side note, I was able to get a Rails app to talk (no sending though, just receiving) to gmail by using the new net/pop library in the latest ruby and making the Mail Fetcher plugin.

But back to the main point, being able to send through the Google Apps account. The technical stuff seems really easy now. Just send your emails out like normal and have some way to check responses either manually or programmatically like I showed above. You're golden, or so it seems.

If you have an application like the one I'm working on, you have many emails being sent for single actions (one to the person who submitted a form, and then 10 - 100 emails being sent to others notifying them of the form submission as well). That seems fine up until you read through the Google Apps help center and find out that you can only send emails to 500 addresses each day per account (so either 1 email message with 500 recipients, or 500 emails each with one recipient).

If you have a commercial application, that is a serious limitation, and one that might stop you from using Google as your email hosting solution.

The important thing to note though is that the limitation is on one account per day. With Google Apps, you receive your Google Apps account which allows you to administer certain things like a calendar, file uploads (which at this point is pretty lame), chatting, and email. Google allows you to create users who can use these features in your domain. So you can create john@mydomain.com and mildred@mydomain.com, and they can have an email account, read your files, chat with each other, and check the mydomain.com calendar. The accounts that get limited are the accounts you create, not your Google App account. There is hope! Google allows you to create quite a few accounts for your domain (200 when you sign up, and you can always ask for more).

What that means is that you are not limited to 500 emails per day, but really 100,000 emails per day (200 * 500). That's a lot of traffic! The obvious way to use all of it is to create a bunch of accounts (don't forget to enable either POP3 or IMAP access!) and randomly send from them all. Just set up account names like "system1@example.com" and "system2@example.com" and randomly choose which one to send from each time you send out an email. It just works, and it distributes the load of sent emails pretty well across all email accounts. Implementation wise, I did something like this.

It ends up sending from email addresses "system100@example.com" through "system110@example.com". Also, in that example I put in the ActionMailer::Base.smpt_settings for clarity, but in reality I have them in the config/mailer.yml file used by the action_mailer_tls plugin.

Hopefully this will help someone who is on the fence about using Google Apps as a solution to email after finding out about the sending limit. Enjoy!

bash aliases

2007-11-03T14:03:00.000-07:00

Bash aliases are such lovely things, so I thought I would share some of my favorites and maybe even get some feedback.

Most of my aliases are there to make software development more enjoyable and less time consuming. As such, most of them deal with the projects I have in my project directories. Here are some of the more interesting ones:


for bigdir in $(ls ~/Developer/Projects/ | cut -d' ' -f7)
do 
  for dir in $(ls ~/Developer/Projects/$bigdir)
    do
      alias cd$dir="cd ~/Developer/Projects/$bigdir/$dir"
      alias m$dir="cd$dir && mate ."
      alias s$dir="cd$dir && ss"
      alias g$dir="~/bin/svngrowl $dir"
      alias o$dir="osascript -e 'tell application \"Terminal\"'
        -e 'tell application \"System Events\"'
        -e 'tell process \"Terminal\" to keystroke \"t\" using command down'
        -e 'tell process \"Terminal\" to keystroke \"t\" using command down'
        -e 'end tell'
        -e 'do script \"s$dir\" in first tab of front window'
        -e 'do script \"m$dir\" in second tab of front window'
        -e 'do script \"g$dir\" in third tab of front window'
        -e 'do script \"localhost\" in second tab of front window'
        -e 'tell second tab of front window to set selected to true'
        -e 'end tell'"
    done
done

alias localhost="open http://localhost:3000"
alias addall="svn st | grep ? | cut -d' ' -f7 | xargs svn add"
alias removeall="svn st | grep ? | cut -d' ' -f7 | xargs rm -Rf"

for var in "server" "generate" "console" "destroy" "plugin"
do
  alias s${var:0:1}="script/$var"
done

The first for loop is probably the most useful set of aliases that I have. It starts off by going through all the projects that live in my various project directories. Then, for each project it gives me a cdproject, mproject, sproject, gproject (very special), and oproject (less special, but still useful).

cdproject

cd's into the project directory

mproject

opens the project in TextMate

sproject

starts a server for the project by calling script/server (which is what the alias ss does in a rails project)

gproject

starts an svngrowl process for the project (which I'll explain later)

oproject

Uses the command line osascript, which invokes applescript, to manipulate Terminal and Firefox

First, an explanation of what oproject is doing. To start, for viewing purposes above I cut the oproject alias into more manageable pieces, but when used should be on one line (as would be the case if it were called from the command line).

This alias is kind of like the Ring of Power, in that it rules all the other aliases, and in the darkness (or the light) binds them... into one convenient call. It opens Terminal, and opens 2 new tabs for a total of 3 tabs (only for Leopard users though :-). It then opens a server in the first tab, opens the project in TextMate in the second tab, starts an svngrowl process in the third tab, goes back to the second tab and runs the alias localhost, and then gives focus back to the second tab (since it seemed to lose focus to the third tab, not sure why). This alias is a huge time saver when you think that every working day I spend about 10 to 15 seconds when I have to open everything up (which adds up to a lot when you have enough work to do!).

As for the gproject alias, this uses a cool little script in my ~/bin directory that starts a loop for the specified project, and every 3 minutes it runs svn up on the project and if there are any changes to the project, it sends out a pretty little growl notification with what was changed, who changed it, the commit message, and if any migrations were committed it changes the priority of the message to high (which for my personal growl settings turns the message text red).

Two scripts are involved in the growling, one bash script, and a ruby script (which is invoked by the bash script). First, the bash script (cut in half for easy viewing):


#!/bin/sh
source ~/.profile
alias | grep cd$@= | awk "{gsub(/cd$@='cd |'/, \"\"); print}" |
awk "{gsub(/~/, \"/Users/`whoami`\"); print}" | xargs ~/bin/svngrowl.rb

And now the ruby script:


#!/usr/bin/env ruby
begin
  directory = ARGV[0]
  project = directory.split("/").last
  ignore = directory.gsub(project, "")

  loop do
    updates = `svn up #{directory}`.gsub(ignore, "").split("\n")
    last_log = `svn log #{directory} --limit 1`
  
    revision = updates.pop.scan(/\d+/)
    committer = last_log.split("\n")[1].split(" | ")[1]
    priority = updates.join("\n").include?("db/migrate") ? 1 : 0
    message = last_log.split("\n")[2..-2].join("\n")
  
    unless updates.empty?
      `growlnotify -s -t "#{project} Update - #{committer}"
      -m "#{message + "\n\n" + updates.join("\n")}" -p #{priority} --image ~/bin/mph.png`
    end
    sleep 180
  end
rescue
  `~/bin/svngrowl.rb #{directory}`
end

All the bash script does is finds the proper directory for the project in question and passes it to the ruby script. The ruby script in turn does an svn up on the directory, collects all the interesting data, and runs growlnotify (the command line growl utility) displaying all of the interesting data. There is also a rescue statement in there that just restarts the process if something goes wrong (an error is thrown if svn can't update properly, which can happen if the network drops out or if you do svn st at the same time that this process is doing svn up). I also used the --image options to pass in a beautiful picture of Mr. Potato Head to use as the icon shown on the notification.

Please feel free to use any of these to make your work/play more productive (except for the picture of Mr. Potato Head... that one is mine!)

sfight!

2007-10-02T03:00:00.000-07:00

I played around yesterday when I got bored and wrote a little script which pits developers on a project against each other to see who has the most subversion commits. It doesn't really mean too much who wins since you can have plenty of little commits that fix things, but it is just fun to do to see who wins! This script assumes the developers on the project have separate subversion logins. Enjoy!


#!/usr/bin/env ruby
developers = ARGV.select {|arg| !arg.empty? && !arg.nil?}
raise "You need two developers to fight" if developers.size < 2

log = `svn log`
separator = log.scan(/^-.*?-$/).first
log_entries = log.split(separator)[1..-2]
names = log_entries.collect {|entry| entry.split(" | ")[1]}

numbers = developers.collect {|developer| names.select {|entry| entry == developer}.size}

results = []
developers.each_with_index {|developer, i| results << "#{numbers[i].to_s.ljust(3)}- #{developer}"}

puts results.sort {|x, y| y.scan(/\d+/).first.to_i <=> x.scan(/\d+/).first.to_i}

Use it like this (assuming you chmod'ed it to be executable, and put it into your ./script directory):


./script/sfight developer1 developer2 developer3

Rails and SSL (https://)

2007-08-28T16:33:00.000-07:00

So I recently had to have some secure routes for a site I am currently working on. If you are using deprec and deploying to a slicehost, getting the SSL certificate stuff working is a snap. If you want your whole site to be over SSL, then all you do is put in a permanent redirect to socket 443 in your app_name.conf file. But, if you need to do something interesting like go back and forth between http and https, then you need to do a little digging.

Doing a google search doesn't bring up anything particularly useful in terms of going back and forth between http and https, but one thing did catch my eye. In your routes you can pass a protocol option to set which protocol you are going to use (something like http, file, svn, https, etc...). I also had to dig through some code to find out how to pass that option in, since it gets set two different ways depending on what kind of route you are using. If you do a map.resource(s) route, then you have to set the option as :requirements => {:protocol => ROUTES_PROTOCOL}, whereas if you just do a map.route_name type route, then you just pass in :protocol => ROUTES_PROTOCOL.

You should also be careful about when you are setting that protocol, as trying to access https://localhost:3000 just doesn't work (such as in development mode). I just set a conditional constant at the top of my routes file like so: ROUTES_PROTOCOL = (ENV["RAILS_ENV"] =~ /development/ ? "http" : "https").

Two more caveats:

First caveat is that if you want to go back and forth between http and https, you MUST explicitly set all of your routes to have http or https protocol. If you don't, then when rails generates your routes it will just use the protocol of the current page. This is annoying, but I can see why they would do this (if you are looking at something securely, then just look at everything securely).

Second caveat is that you must use the route_name_url throughout your application instead of route_name_path, as some are accustomed to. The _url named route gives you the full url (duh), which is what you want if you are on a page using http and want to go to https (whereas _path only gives you the uri, and will use the protocol and host of the current page). If you already use _url, then you are one step ahead in the game.

Here is a sample of some of my routes using the protocol option:


# Public routes using http protocol
map.with_options :protocol => "http" do |http|
  http.login '/login',   :controller => 'sessions', :action => 'new'
  http.logout '/logout',  :controller => 'sessions', :action => 'destroy'
  http.resources :users, :sessions, :requirements => {:protocol => "http"}
end

# Public routes using https protocol
map.with_options :protocol => ROUTES_PROTOCOL do |https|
  https.form_step '/form/*slug', :controller => 'forms', :action => 'show'
  https.form_edit '/edit/*path', :controller => 'forms', :action => 'edit'
  https.resources :forms, :requirements => {:protocol => ROUTES_PROTOCOL}
end

Different ways to program

2007-05-23T17:40:00.000-07:00

I've been told that there are different ways to program. One way is procedural, where you work with something (let's say a shopping cart), and it does one thing, then it does another, then another, and so on until oblivion. Then there is object-oriented programming, where you have objects that you deal with (like in real life!) and not just random things that you happen to write code about. So your shopping cart has attributes, like it can hold items, you can remove items from it, and when it is time to checkout you remove all of the items from the cart and pay for the items. And just like at the grocery store, you leave the cart at the store (hopefully).

I guess the difference programming wise is that procedural programming you just write a whole slew of functions that can be applied anywhere and it is the programmers job to decide when and where he should use them, whereas in OOP you have classes for your objects, and each class has specific functions to deal mainly with instances (objects) of the class. I'm sure you can have the guise of OOP and still be doing procedural programming.

I only mention all this because I was looking through a website that the client I am working for said to look at and try to mimic what they were doing with some javascript showing and hiding. It all boiled down to that they were toggling the CSS display property on DOM elements between "block" and "none" (the show() and hide() functions in Prototype). Well, I finally found that out after looking on in amazement at a 1211 line function that seemed to show and hide everything on the entire website! Notice that I said function. They just used this massive catch all function to do all the showing and hiding, instead of trying to somehow abstract it into some showing/hiding class thingy (don't ask me how I would do it though, because I have no idea). Maybe I would have done it like that if I wouldn't have gone insane trying to remember exactly where I was in the 1200+ lines of code.

And as a side note, look at this code:



if (getFormObject("reslease","taxes")[0].checked) {
   setVisDisp('taxesDiv',true);
   if (getFormObject("reslease","taxesTenant")[0].checked) {
      setVisDisp('taxesTenantDescDiv',true);
   } else {
      setVisDisp('taxesTenantDescDiv',false);
   }

   if (getFormObject("reslease","taxesGovt")[0].checked) {
      setVisDisp('taxesGovtDescDiv',true);
   } else {
      setVisDisp('taxesGovtDescDiv',false);
   }

} else {
   setVisDisp('taxesDiv',false);
}

That was from line 1070 to 1086. After almost 1100 lines of madness, how do really know that you want the first element from the "reslease" form? And what is a "taxesTenantDescDiv"? My mentor has been drilling it into my head to make whatever your coding self-descriptive, so that when you go back you never have to wonder what the "taxesTenantDescDiv" is, or where it lives.

Regular Expression Finding in TextMate

2007-04-25T04:27:00.000-07:00

Recently I had to convert a list of items in a word doc over to a yaml file in order so that I could load the data into our database. The word doc only contained the data necessary that a human could read and understand what was needed (meta data), but looked nothing like what the final yaml needed to look like. So I had two options, either sit there for 2 hours typing everything by hand over to yaml, or spend 20 minutes learning how to use the regular expression finding feature in TextMate. I went with the latter and am oh so glad I did. $1 is now my friend.

If you haven't tried it yet, play around with it ad see how awesome it can be!

Selenium Remote Control Plugin!

2007-04-15T04:01:00.000-07:00

I recently had a need to get the selenium testing working for another app I was working on and had to copy a bunch of files and whatnot, which I knew was bad. So I took about an hour the other morning and turned it into a plugin for all the world to enjoy.

Here's the repository. Enjoy!

ActiveRecord magic

2007-03-14T19:18:00.000-07:00

Or so Josh Susser puts it. I think it is. It's dynamic finding. I had no idea this was possible until a few days ago when i wrote some moderately-ugly looking method like:


def find_or_create_order
  unless @order = Order.find(params[:blah])
    @order = Order.create!(params[:blah])
  end
end

Then one of the geniuses (since I am working with multiple geniuses) I am working with right now came over and said "You can just use Order.find_or_create_by_blah(params[:blah])." I tried it out, and to my amazement (and I am easily amazed, and amused) it worked! I could try to explain it, but I think Josh Susser has a far greater understanding of how it works, so just go read his post if you're interested.

I did however do some playing, and found some really cool things you can do with these finders. Let's say you have an ActiveRecord model named Order. Let's also say that Order has 7 attributes on it, each labeled "1" through "7". You can write a massive line of code to find or create an Order as follows:


Order.find_or_create_by_1_and_2_and_3_and_4_and_5_and_6
_and_7("1", "2", "3", "4", "5", "6", "7")

The possibilities are endless!!!! Or so I imagine. One easy way to find out is to make an ActiveRecord model with 50 attributes and try to find_or_create one using the above method :-) I'm sure there are more sophisticated ways of using this, but just think of the shock value when you show this to a competent programmer. Their jaw will drop, and they'll say something like "You're smoking crack, man!"

Selenium and Ruby: They actually work together!

2007-02-14T19:27:00.000-08:00

So we found out this week. Just wanted to do a brain-dump so I could remember how to configure the Selenium Remote Control to work with Ruby on Rails, and hopefully impart knowledge to those who wish to play with Selenium.

For a refresher, Selenium is a testing framework that allows you to test the clicking-around aspect of your web application. It allows you to record clicks and all sorts of actions that you may routinely perform to test whether you have broken something in your web application after changing some code. I recently started using this week after having some major changes in a application that I am working on with a friend.

A major portion of code was changed from being mostly server-side code (with a ton of SQL calls) to client side (using JavaScript). This threw a wrench into our testing of the application because we could no longer test the actions that one might perform on the screen because there were very few calls being made to the server through Rails (it was all being done with JavaScript in the browser). So to test this we decided to go with the most excellent program Selenium. The following is a somewhat detailed overview of how to get Selenium tests working with Ruby and Rails, and by that I mean being able to write your Selenium tests with Ruby code:

Get Selenium IDE (the Firefox extension).

Download the Selenium Remote Control and stick the unzipped folder into the lib directory of your application; you may wish to delete all the extraneous garbage, since all you will need is the ruby folder and the server folder. Also, in your lib directory make a file named selenium.rb, and the only thing needed in the file is require "selenium/ruby/selenium-ruby". You will see why later.

Start a test server (script/server -e test --port 3001) for selenium to work with so it doesn't try to use your development db. Choosing the port number is optional, but you will need to know which port this server is running on to use in your test file.

Start a java server (make sure you have JRE 1.5 or higher) by navigating to the selenium/server folder in your lib directory on a command line and type java -jar selenium-server.jar. This will start a Jetty server on port 4444.

Make a folder in your test directory named selenium. This is where you will put your selenium tests when you make them.

Save some recorded tests from the Selenium IDE in your test/selenium folder. You will need to perform the test in the IDE and then export the tests as "Ruby - Selenium RC". This will package the normal table tests into some handy Ruby code. You will need to rename the test class to be a proper name (something like MyNewSeleniumTest) and also whack the require 'test/unit' and replace it with require File.dirname(__FILE__) + '/../test_helper'.

Your selenium test should look something like the following, and should follow the normal Rails test-naming conventions "my_new_test.rb":


require File.dirname(__FILE__) + '/../test_helper'
require "selenium"

class MyNewSeleniumTest < Test::Unit::TestCase
  def setup
    @verification_errors = []
    if $selenium
      @selenium = $selenium
    else
      @selenium = Selenium::SeleneseInterpreter.new("localhost", 
        4444, "*firefox", "http://localhost:3001", 10000);
      @selenium.start
    end
    @selenium.set_context("test_new", "info")
  end
  
  def teardown
    @selenium.stop unless $selenium
    assert_equal [], @verification_errors
  end
  
  def test_new
    @selenium.open "/"
    @selenium.type "b-query", "hey yo"
    @selenium.click "b-searchbtn"
    @selenium.wait_for_page_to_load "30000"
    @selenium.click "link=Privacy Policy"
    @selenium.wait_for_page_to_load "30000"
  end
end

The line require "selenium" will call the selenium.rb file in your lib folder, which calls the selenium-ruby.rb file. This nice trick avoids having to load and use the selenium gem. You can, however, download the gem if you wish (gem install selenium) and kill the selenium.rb in your lib folder, and everything will still work properly.

Pay close attention to this line in the setup method: @selenium = Selenium::SeleneseInterpreter.new("localhost", 4444, "*firefox", "http://localhost:3001", 10000). The "http://localhost:3001" should be the port that your test server is running on. We banged our heads against a wall for quite a while until we figured that out. You can also choose Firefox or (::dread::) IE as the browser that Seleium will open. Mac users may want Safari, but we haven't checked to see about any surprises with Safari and Selenium.

That should be all you need (not that much, right?). Now you can run your tests from the command line like you would any other individual test (in Win-doze: ruby test\selenium\new_selenium_test). You can set up a rake task (to go into your lib/tasks folder) to run all of your tests for you pretty easily, like the following one we set up:


namespace :test do
  Rake::TestTask.new(:selenium => "db:test:prepare") do |t|
    t.libs << "test"
    t.pattern = 'test/selenium/**/*_test.rb'
    t.verbose = true
  end
  Rake::Task['test:selenium'].comment = 
    "Run the selenium tests in test/selenium"
end

Now you can simply call rake test:selenium from a command line (with your test server and Jetty server running) and your tests will pop up in a new browser window and spit out the results at the end. If you do run any of the tests individually without using the rake task you will be faced with the problem of your test db not clearing out (for some unknown reason Rails does not seem to want to treat the Selenium tests as normal tests, and thus does not prepare the test database before running each test). We struggled with this for quite a while and had to finally dig into the rake tasks to find out how to tell MYSQL what we wanted it to do. The result was a small smorgasbord of ActiveRecord::Base commands to make a schema of the current test db, clear the db completely, and then rebuild the db using the just made schema. The following code was put into a test helper (which was required by the test file by using the test_helper :selenium call) and called inside of the setup of the test:


def prepare_test_db    
  abcs = ActiveRecord::Base.configurations
  ActiveRecord::Base.establish_connection(abcs[RAILS_ENV])
  File.open("#{RAILS_ROOT}/db/#{RAILS_ENV}_structure.sql", "w+"
    ) { |f| f << ActiveRecord::Base.connection.structure_dump }
  ActiveRecord::Base.connection.recreate_database(abcs["test"]["database"])
  ActiveRecord::Base.establish_connection(:test)
  ActiveRecord::Base.connection.execute('SET foreign_key_checks = 0')
  IO.readlines("#{RAILS_ROOT}/db/#{RAILS_ENV}_structure.sql"
    ).join.split("\n\n").each do |table|
      ActiveRecord::Base.connection.execute(table)
  end
end

Now you're test db will be cleared out no matter how you invoke the test.

Now, go and use this information for the greater good. Share it and love it.