improve bench.rb

[jordan.breeding@…]

Currently bench.rb has a few inconsistencies (number printed to screen is not the number of times looped) and doesn't work well under ruby 1.8 or 1.9.

Having it work on multiple versons of ruby is useful for benchmarking regressions against other ruby versions.

I am including a patch I use locally to use able to run bench.rb under all ruby versions (by catching errors).

diff --git a/bench.rb b/bench.rb
index e05833b..2c47c62 100644
--- a/bench.rb
+++ b/bench.rb
@@ -7,6 +7,16 @@
 # The same script can also be executed using /usr/bin/ruby or ruby19 to 
 # compare the implementations.
 
+def executeWithErrorHandling
+  if (block_given?)
+    begin
+      yield
+    rescue Exception => e
+      print(" ************* BENCHMARK FAILED ************* ")
+    end
+  end
+end
+
 def fib(n)
   if n < 3
     1
@@ -80,168 +90,248 @@ Benchmark.bm(30) do |bm|
 
   # Fixnum arithmetic.
   bm.report('10 fib(30)') do
-    i=0; while i<10; fib(30); i+=1; end
+    executeWithErrorHandling do
+      i=0; while i<10; fib(30); i+=1; end
+    end
   end
   bm.report('10 fib(35)') do
-    i=0; while i<10; fib(35); i+=1; end
+    executeWithErrorHandling do
+      i=0; while i<10; fib(35); i+=1; end
+    end
+  end
+  bm.report('tak') do
+    executeWithErrorHandling do
+      tak(18,9,0)
+    end
+  end
+  bm.report('tarai') do
+    executeWithErrorHandling do
+      tarai(12,6,0)
+    end
   end
-  bm.report('tak') { tak(18,9,0) }
-  bm.report('tarai') { tarai(12,6,0) }
-  if RUBY_VERSION.to_f > 1.8
-    # Ruby 1.8 is too weak for this benchmark.
-    bm.report('ackermann') { ack(3,9) }
+  bm.report('ackermann') do
+    executeWithErrorHandling do
+      ack(3,9)
+    end
   end
 
   # Loops.
   bm.report('10000000 times loop') do
-    3000000.times {}
+    executeWithErrorHandling do
+      10000000.times {}
+    end
   end 
   bm.report('30000000 times loop') do
-    30000000.times {}
+    executeWithErrorHandling do
+      30000000.times {}
+    end
   end 
   bm.report('10000000 while loop') do
-    i=0; while i<10000000; i+=1; end
+    executeWithErrorHandling do
+      i=0; while i<10000000; i+=1; end
+    end
   end 
   bm.report('60000000 while loop') do
-    i=0; while i<60000000; i+=1; end
+    executeWithErrorHandling do
+      i=0; while i<60000000; i+=1; end
+    end
   end
 
   # Messages.
   bm.report('30000000 msg w/ 0 arg') do
-    o = Class1.new
-    i=0; while i<10000000; o.method1; o.method1; o.method1; i+=1; end 
+    executeWithErrorHandling do
+      o = Class1.new
+      i=0; while i<30000000; o.method1; o.method1; o.method1; i+=1; end
+    end
   end
   bm.report('30000000 msg w/ 1 arg') do
-    o = Class1.new
-    i=0; while i<10000000; o.method2(i); o.method2(i); o.method2(i); i+=1; end 
+    executeWithErrorHandling do
+      o = Class1.new
+      i=0; while i<30000000; o.method2(i); o.method2(i); o.method2(i); i+=1; end
+    end
   end
   bm.report('10000000 super w/ 0 arg') do
-    o = Class2.new
-    i=0; while i<10000000; o.method1; i+=1; end
+    executeWithErrorHandling do
+      o = Class2.new
+      i=0; while i<10000000; o.method1; i+=1; end
+    end 
   end
   bm.report('10000000 super w/ 1 arg') do
-    o = Class2.new
-    i=0; while i<10000000; o.method2(i); i+=1; end
+    executeWithErrorHandling do
+      o = Class2.new
+      i=0; while i<10000000; o.method2(i); i+=1; end
+    end
   end
   bm.report('10000000 #send') do
-    o = Class1.new
-    i=0; while i<10000000; o.send(:method1); i+=1; end
+    executeWithErrorHandling do
+      o = Class1.new
+      i=0; while i<10000000; o.send(:method1); i+=1; end
+    end
   end
   bm.report('30000000 tail calls') do
-    tail(30000000) 
+    executeWithErrorHandling do
+      tail(30000000)
+    end
   end
 
   # Instance variables.
-  bm.report('10000000 ivar read') { bench_ivar_read(10000000) }
-  bm.report('30000000 ivar read') { bench_ivar_read(30000000) }
-  bm.report('10000000 ivar write') { bench_ivar_write(10000000) }
-  bm.report('30000000 ivar write') { bench_ivar_write(30000000) }
+  bm.report('10000000 ivar read') do
+    executeWithErrorHandling do
+      bench_ivar_read(10000000)
+    end
+  end
+  bm.report('30000000 ivar read') do
+    executeWithErrorHandling do
+      bench_ivar_read(30000000)
+    end
+  end
+  bm.report('10000000 ivar write') do
+    executeWithErrorHandling do
+      bench_ivar_write(10000000)
+    end
+  end
+  bm.report('30000000 ivar write') do
+    executeWithErrorHandling do
+      bench_ivar_write(30000000)
+    end
+  end
 
   # Const lookup.
   bm.report('30000000 const') do
-    i=0; while i<30000000; i+=ConstantOne; end
+    executeWithErrorHandling do
+      i=0; while i<30000000; i+=ConstantOne; end
+    end
   end
 
   # Blocks
   bm.report('30000000 yield') do
-    o = Class1.new
-    o.yield1(30000000) {}
+    executeWithErrorHandling do
+      o = Class1.new
+      o.yield1(30000000) {}
+    end
   end
   bm.report('30000000 msg w/ block+yield') do
-    o = Class1.new
-    i=0; while i<30000000; o.yield2 {}; i+=1; end
+    executeWithErrorHandling do
+      o = Class1.new
+      i=0; while i<30000000; o.yield2 {}; i+=1; end
+    end
   end
   bm.report('30000000 Proc#call') do
-    o = proc {}
-    i=0; while i<30000000; o.call; i+=1; end
+    executeWithErrorHandling do
+      o = proc {}
+      i=0; while i<30000000; o.call; i+=1; end
+    end
   end
   bm.report('30000000 dvar write') do
-    i=0
-    30000000.times { i=1 }
+    executeWithErrorHandling do
+      i=0
+      30000000.times { i=1 }
+    end
   end
 
   # Eval
   bm.report('1000 eval') do
-    i=0
-    s = "#{1+1}+#{20+20}"
-    while i<1000
-      eval(s)
-      i+=1
+    executeWithErrorHandling do
+      i=0
+      s = "#{1+1}+#{20+20}"
+      while i<1000
+        eval(s)
+        i+=1
+      end
     end
   end
   bm.report('30000000 binding-var write') do
-    i=0
-    eval('while i<30000000; i+=1; end')
+    executeWithErrorHandling do
+      i=0
+      eval('while i<30000000; i+=1; end')
+    end
   end
 
   # Break
   bm.report('30000000 while break') do
-    i=0; while i<30000000; while true; i+=1; break; end; end
+    executeWithErrorHandling do
+      i=0; while i<30000000; while true; i+=1; break; end; end
+    end
   end
   bm.report('10000000 block break') do
-    i=0; while i<10000000; 1.times { i+=1; break }; end
+    executeWithErrorHandling do
+      i=0; while i<10000000; 1.times { i+=1; break }; end
+    end
   end
 
   # Next
   bm.report('30000000 while next') do
-    i=0; while i<30000000; i+=1; next; exit; end
+    executeWithErrorHandling do
+      i=0; while i<30000000; i+=1; next; exit; end
+    end
   end
   bm.report('10000000 block next') do
-    i=0; while i<10000000; 1.times { i+=1; next; exit; }; end
+    executeWithErrorHandling do
+      i=0; while i<10000000; 1.times { i+=1; next; exit; }; end
+    end
   end
 
   # Exception handlers.
   bm.report('60000000 begin w/o exception') do
-    i=0
-    while i<60000000
-      begin
-        i+=1
-      rescue
+    executeWithErrorHandling do
+      i=0
+      while i<60000000
+        begin
+          i+=1
+        rescue
+        end
       end
     end
   end
   bm.report('60000000 ensure w/o exception') do
-    i=0
-    while i<60000000
-      begin
+    executeWithErrorHandling do
+      i=0
+      while i<60000000
         begin
+          begin
+          ensure
+            i+=1
+          end
         ensure
           i+=1
         end
-      ensure
-        i+=1
       end
     end
   end
   bm.report('50000 raise') do
-    i=0
-    while i<50000
-      begin
-        raise
-      rescue
-        i+=1
+    executeWithErrorHandling do
+      i=0
+      while i<50000
+        begin
+          raise
+        rescue
+          i+=1
+        end
       end
     end
   end
 
   # Method
   bm.report('3000000 Method#call w/ 0 arg') do
-    o = Class1.new
-    m = o.method(:method1)
-    i=0
-    while i<3000000
-      m.call
-      i+=1
+    executeWithErrorHandling do
+      o = Class1.new
+      m = o.method(:method1)
+      i=0
+      while i<3000000
+        m.call
+        i+=1
+      end
     end
   end
   bm.report('3000000 Method#call w/ 1 arg') do
-    o = Class1.new
-    m = o.method(:method2)
-    i=0
-    while i<3000000
-      m.call(i)
-      i+=1
+    executeWithErrorHandling do
+      o = Class1.new
+      m = o.method(:method2)
+      i=0
+      while i<3000000
+        m.call(i)
+        i+=1
+      end
     end
   end
-
 end

[jordan.breeding@…]

Patch for bench.rb as an attachment.

[lsansonetti@…]

Thanks for this work, but I think it's preferable to get rid of bench.rb and replace it with something more elaborate instead, which would try to micro-benchmark most of the core classes/methods. We would use this as a regression test suite for the future. Matt started working on this, if you're interested to help you might want to ping him :)

[jordan.breeding@…]

I might do that soon, I remembered Patrick saying something about that work, but I never saw anything go by so I didn't know if it was stalled or canceled or what. I have no problem with closing this given that, I just wanted to get this patch somewhere other than my hard drive.

[mattaimonetti@…]

Hey Jordan,

Feel free to contact me directly if you are interested in working on the micro-benchmark/regression suite project (aka HERBS, Hopefully Exhaustive Ruby Benchmark Suite). I just started a few days ago so we can see the progress made by the latest IO improvement.

Thanks,

- Matt

[jordan.breeding@…]

Replying to mattaimonetti@…:

Hey Jordan, Feel free to contact me directly if you are interested in working on the micro-benchmark/regression suite project (aka HERBS, Hopefully Exhaustive Ruby Benchmark Suite). I just started a few days ago so we can see the progress made by the latest IO improvement. Thanks, - Matt

Sounds good. I don't start my new job until Oct. 1. I am working on a Ruby based project that I have been working on over the summer still, but I could probably contribute a little bit to this as well.

I am currently trying to come up with a short, reproducible test case for why a program of mine that works in 1.8/1.9 doesn't work in MacRuby at all.

Feel free to drop me a line at jordan (dot) breeding (at) me (dot) com.