Category: Tech

mtnwestruby: RubyCLR and Ruby.NET by John Lam
17 March 2007

John works for Microsoft. Before coming to Microsoft, he built the bridge for running Ruby on the CLR (common language runtime).

Why would anyone want to run Ruby on someone else’s virtual machine? To answer
that, we look at the overlap between what we want to do and what we actually
do at work. It would be nice to maximize that overlap. Many of us want to use
dynamic languages, by many of us have to use .NET or Java. Running dynamic
languages on the one of these VMs helps maximize the overlap.

The interop layer for Python is going to be fundamentally different than that
for Ruby so that when you write a Python program using .NET libraries, it still
feels like Python code, and when you write Ruby code, using .NET libraries, it
still feels like writing Ruby code.

Lutz is an MS employee that worked on Intentional Programming, which never went
anywhere at MS. He created an editor that would parse the AST (abstract syntax
tree) and display it in different ways. This idea lives on in the .NET
Reflector, which will decompile .NET assemblies into IL, Visual Basic, C#,
Delphi, etc. It’s interesting to use Iron Python to compile code, and use the
.NET Reflector to look at the IL.

Dynamic methods are an interesting feature of the CLR in version 2.0. If they’re not going to be used, they get garbage collected. Before 2.0, once you load code, it stays loaded for the lifetime of the VM.

Interesting: The CLR supports polymorphic methods based on the return type, not jus the arguments. This isn’t a feature that C# supports.

The Ruby to .NET interop is extremely fast — millions of operations per
second.

John used Vim on Windows when he edited code.

On April 30th at the MS mixed conference, we will see the interesting things
that John Lam has been working on since he joined Microsoft, which he can’t
talk about right now. However, the really interesting conference will be Microsoft PDC because the technology will be more mature.

Q: Can you run Rails on the Ruby CLR?
A: No idea.

Q: Can I run Ruby.NET on mono or on my PowerPC?
A: There’s nothing that would preclude it from running on Mono, but Microsoft is not working on it.

Mountain West Ruby Conference: Lightning Sessions — Five minutes each.

16 March 2007

Mike. BinaryLottery uses ‘figlet’ to translate strings to ASCII art.

WAX by dan@eparklabs.com. WAX – Web Application X — A similar thing to Rails, but a totally different concept. Strongly tied to the database. Extremely fast development for enterprise database applications. Used by 10% of Fortune 500 companies, although they don’t know that. Self-contained applications. Designed with CSS. Website/CMS and Applications are integrated. REST architecture. Start with “wax on”. Stop with “wax off”. Allows you to GET and POST Excel spreadsheet files, among other things.

Managing SSH keys with Capistrano by Jade Meskill from Pheonix, AZ. Capistrano takes all developer’s SSH public keys, and puts them into an authorized keys file on the remote boxes they need to deploy software to. http://iamruious.com

Goldberg presented by Coby Rhenquist. He found that Rails leaves developers to themselves after getting their application framework stetup. Then he found the Goldberg generator, which gives you a more complete site, including better looks, a login page, users, roles, permissions and built-in CMS (textile), etc. Gives the administrator the capability to wire things up using menus in the web UI front end. Goldberg.rubyforge.org. Disadvantage: When rails 1.2 comes out, you have to re-do your app.

CruiseControl.rb 1.0 was just released. http://cruiseControlrb.thoughtworks.com It should take about 10 minutes to download and install. No XML or XSLT as with Java CruiseControl. Not released as a gem.

JRuby deployment. Charles Nutter. Just released JRuby Rails Integration.

$ rake war:standalone:create

JRuby has better database support than Rails because of JDBC.

LogWatchR by Pat Eyler (uses Suse Linux and vim). Concepts: Simple, Extensible, Low Support costs. Uses YAML to specify good and bad patterns. LogWatchR saves statistics to that YAML file as well. Written in 250 lines of Ruby. Handles 2250 log entries/sec. Likes Ruby because it’s easy to write, easy to read, easy to maintain.

Mountain West Ruby Conference: SBN
Tag: mtnwestruby
16 March 2007

James Britt – Black-boxing with Ruby

Reinventing the wheel is overrated. Why not reuse what other people have already implemented, even from other languages?

Trac: A good project tracking system. Requires too much use of the mouse.
What if: I could use Trac from the command line? Tracula.

Tracula is written in Ruby and uses Hpricot & Mechanize to pretend to be a web browser. Unfortunately, tracula is brittle because the web page UI tends to change — it’s not an API they’re publishing. Many features are missing from Tracula, but that’s ok — he writes the functionality as he needs it, and stubs out missing behavior with hardcoded values.

Tracula acts as a proxy, so the code using tracula doesn’t break just because trac changes.

WordPress: A best of breed web log system.
What if: I could have the great comment system for the Django Book?
www.djangobook.com. Uses Yahoo UI and Yahoo.EXT.
Comet is written in Ruby. Comments made by users are redirected via Apache modprox through comet, which takes the comment, and posts it to WordPress.

Lessons learned:

• WordPress plugins are the way to go.
• Repurposing other people’s code can bite your hard.
• Most web sites offer APIs, including TRAC. Use them, but still use a proxy layer so your software is insulated from changes in the API.
• Proxies can add exception handling that is missing from APIs.

Techniques:

• Screen scraping
• DOM munging
• Proxies, proxies, proxies!
• Save off a copy of web pages while you are developing your HTML screen scraper, and hit those saved pages so that you don’t run up someone’s bandwidth bill.

Mountain West Ruby Conference: Ruby Implementor’s Panel
16 March 2007

Ruby on Parrot has a JIT and garbage collection. Needs to be polished. Has an interop convetion so languages can call into each other. Bytecode runs on most common hardward — Intel, PPC.

John Lam – CLR Ruby: CLR naming conventions are nothing like Ruby naming conventions, but you want to make the interop look like Ruby. CLR was originally envioned to be a static language runtime, so there are similar problems as for the JRuby guys. The CLR platform gives you some things “for free”, like debugging support.

Edwin Pheonix – Rubiness: Rubiness is designed new, from the ground up, and meant for Ruby. The idea is to keep it small and simple while supporting full Ruby. Goal is to hit 1.0 by October. It will be 1.8.5 compliant and support Rails 1.2.

Charles Nutter & Tom Enebo – JRuby: Will release 1.0 in the next couple of months. Goal is to get it faster.

Question and Answer Session

Performance

• Nutter: The language shootout posted recently was pretty good. JRuby is about twice as slow as Ruby, although some things are starting to run faster. When compiled, JRuby runs faster than Ruby.
• Ty: Slowly seeing more performance as they work on it.
• Pheonix: Rubiness is slow and makes the other implementations look awesome. Up to this point, he has done no performance tuning. Since the language shootout, he’s put some time into it. He was beating Ruby 1.8.5 on some of the benchmarks.
• Lam: Iron Python is the fastest python out there, depending on who’s benchmarks you believe. This proves that using the CLR is viable. Compile chunks of dynamic code, and garbage collect them later.
• Parrot Guy: Parrot is able to beat C code in some cases, similar to the JVM + Hotspot. Speedups as much as 1.5x over Perl 5, but it depends on what you’re doing.

What features of the Ruby language are the most problematic?

• Pheonix: Continuations are one of the easiest. Dynamic dispatch is a pain, but it’s a fundamental language feature. Eval.
• JRuby: Emulating Ruby’s threading model. Eval is a pain, which is why they have a mixed mode interpreter. Compatibility with esoteric features —
• Pheonix: An esoteric feature is like what you can take in as a block argument.
• Ty: How many people have passed Nil as a block argument? ….
• Pheonix & Nutter: Who here has used ‘retry’ from a block? (one person). Do you know what it does? (no). It re-runs the code.
• Parrot Guy: The dynamic nature of the language makes it difficult.

What advantage does your VM give?

• Parrot Guy: Interop with already written libraries — SQL, etc.
• Lam: You get Profiling, Debugging, Code coverage for free. Auto-complete in IDEs.
• Pheonix: The possibilities are wide open right now. Re-write the byte-code on-the-fly if you want. His goal is to build a platform for other people to get their work done — an extensible VM.
• JRuby (Nutter): Interop with Java libraries. Most companies already have and use Java, so there is much to leverage.
• Ty: How many people have trouble getting Ruby deployed in their environment? (No one — everyone here uses Ruby already)
• Pheonix: Deployment is a big issue. How do we distribute applications without having to ask people to install a VM?

How do your implementations deal with Unicode?

• JRuby: JRuby can use the Rails multi-byte library natively, or uses the Java unicode support.
• Pheonix: Rubiness doesn’t have Unicode, but he’d like it to.
• Parrot Guy: Parrot supports unicode. It’s not that hard to make Unicode work most of the time, today.

Is there a website for a formal specification of Ruby?

• Nutter: I’m still running the website in my basement, and sometimes people contribute to it. http://headius.com/rubyspec — it’s a community spec for Ruby. Few people are clamoring for a spec, except for the people doing alternative implementations.

Is there a common benchmark suite?

• Pheonix: Believes that each implementation will have their won benchmark suites. However, they will run each other’s benchmark suites.
• Nutter: We’re running the test suites that have been written so far, including Rubycon. See the “ruby test” project on Ruby forge.
• Pat Eyler: Various gems will automatically be run against the different VM implementations, and results submitted to the VM implementors.

Why not compile to C code?

• Pheonix: There are many things in Ruby that don’t map to C. Two or three projects to this end have stalled.
• Lam: Iron Python compiles down to static assemblies, so it’s possible to go from Ruby to assembly code, although it still leverages the CLR libraries, garbage collector, etc.
• Nutter: Why re-invent the wheel of a garbage collector, the VM, boundary between libraries? People have worked hard to implement general purpose VMs, so let’s leverage their work.
• Lam: Interpretation isn’t as horrible as it seems. (I overheard him say: Slow application startup time is a problem. Microsoft employs a lot of lisp refugees.)

Mountain West Ruby Conference: SBN
16 March 2007

Carl Youngblood. Simple Bayesian Networks with Ruby.

How do we handle uncertainty? Probablilty theory.
How do we come up with probabilities? From experience, from beliefs — but either of these may not be accurate because of too few samples or skewed samples.


gem install sbn


The Ruby version is faster than his C++ version. Premature optimization is the root of all evil. — Donald Knuth

xmlbif – the format that the bayes network is stored to and read from. Compatible with the Java Bayes package. See http://en.wikipedia.org/wiki/Bayesian_network

He couldn’t get his SBN demonstration program working as expected. Sounds like SBN isn’t ready for prime time. Future improvements:

• Exact inference
• True continuous variables
• Etc.

http://www.pixelglow.com/macstl/

Q: Why not do this in R, and connect R to Ruby?
A: Good idea.

I think it would be cool if someone showed how to connect SBN into CruiseControl.rb to identify what patterns of development and which developers are most likely to break a build.

Mountain West Ruby Conference by Gregory Brown
16 March 2007
Pragmatic Community Driven Development in Ruby. Or “Rolling the dice with Ruport”. “Reporting Sucks”.

Gregory spent several minutes at the beginning of the presentation trying to get his Nvidia X display settings worked out for both the projector and his screen. None of the other people using Linux desktops had the same problems, and neither did the people using Windows or Mac OS X.

Ruport couldn’t have happened without a community to support it. Community = people + problems. Why is community important? Community helps root out bad ideas, and we all have bad ideas. With a good community, you can discover bad ideas sooner, and replace them with better ideas (hopefully).

How do we leverage community? One aspect is to pick a license that encourages people to get involved.

What license do we choose? Licenses aren’t as easy to “refactor” as code. Choose a license that is “right” for your project. Don’t write your own. In order to be taken seriously at a community and at a corporate level, pick one of the already approved OSS licenses. Picking a license is a compromise.

BSD/MIT favors individuals.
GPL favors communities.

Reporting is a big domain. Sometimes, it’s necessary to integrate with software with less than ideal licensing terms. The Ruport community chose the Ruby License.

Communities are idea warehouses.

You can only really work passionately on your problems. Find people who have the same problems. Learn to say “no”. Sometimes, less is more – you’ve got so many ideas floating around and such large groups – sometimes you need smaller groups and a focused, trimmed down set of ideas so that you can define software and implement it. It’s easier to read tens of emails per week than thousands.

Discussion led to code, which led to bug reports.

Mailing lists are not a bug tracker. Gforge is, but it does too much. They decided to use SVN + Trac for Ruport. Allows a casual user to subscribe to what’s happening and to contribute. It’s amazing how much good tools can affect a community and allow contributions. But what about RubyForge? Virtualy all interesting Ruby software is on RubyForge, which gives your software exposure. They used svk and cron to mirror their repository over to RubyForge.

Friction affects contribution. Explain how people can contribute bug reports, patches, etc.

Every patch is valuable. Not every patch is suitable, but the relationships you can establish are important. Patches (suitable or not) give you an idea of what people are trying to get out of your software. Still want relevant patches. What is relevant? It took them 15 months to figure it out – way to long. They came up with a roadmap and a scope definition. They didn’t just scope features, but the design as well.

We have to be careful not to accept code/features that are only for ourselves – that will never be used by anyone else. They moved some things out of the core into plugins.

Unique project identity is good. Half-way implemented features are bad for users.

Recommends the book “Producing OSS Software” by Karl Fogel, which is available for free online.

Ruport 1.0 will be released on May 15, 2007.

Sometimes, developers have to “hide” from the large community surrounding their project, simply because they don’t have time to respond to all questions and ideas. They have to make time for families, work, and for development. It’s good to get smart users involved to help answer questions.

Mountain West Ruby Conference
Ruby Queues (RQ) by Ara Howard
16 March 2007

http://www.linuxjournal.com/article/7922
Ara works for NOAA — primarily with satellite data sets. 50KLOC, all paid for by tax payer dollars. Builds medium sized 10-20 node distributed systems.

RQ helps build instant distributed linux clusters. When presenting RQ to scientists, he rarely mentions Ruby. Today, he will talk about the technical side of RQ. RQ isn’t one of the most interesting pieces of software he’s written, but he learned more than average while writing it. One of the reasons he teaches and presents is because he learns while doing it.

RQ has been used to help generate power outage maps after hurricanes hit. Why did he develop it? The lab purchased a bunch of linux machines instead of a Cray, because it was cheaper. His job was to make them work together. He tends to believe that the first link on Goggle will yield the information he needs, so he went looking for a simple distributed computing framework. The solutions he found were the wrong fit, or overly heavyweight. In their environment, the programs that act on the data follow the data, because it’s more expensive to move data than to move programs. He decided to write RQ.

Tried using MySQL for the server queue controller. However, it adds complication with setting up usernames and passwords, and getting approval for the security thereof. He decided to leverage what was already approved – traditional UNIX file permissions and NFS for shared access to data. He also couldn’t run a process as root, or have it listen on a TCP port.

Needed NFS-safe lock files.

gem install lockfile # he wrote this package

NFS lockd wasn’t very good at throughput or fairness. One node would get the lock 500 times in a row, then the next node 500 times, etc. He wrote lock-polling code with a back-off algorithm. It took a while to get it right.

Ended up using SQLite for the shared data store. “Beats the pants off pstore, fsdb, Madeline, etc.” Most of these un-ideal solutions didn’t work well with NFS-heavily-cached data. They would run for 2 weeks, then get corruption. In contrast, SQLite is very robust over NFS — it detects and recovers from corruption.

gem install slave

How does a normal user install daemon processes? RQ cron.

nrtq query – input and output in YAML. He didn’t tell scientists that it was YAML. They didn’t need to know. Using YAML meant he didn’t have to write his own parser, and it’s human readable.
RQ is being used on a single host to queue jobs. There’s a Rails plugin.

Lessons learned:

• NFS is quirky, but it’s the defacto standard. We get to live with it and work around the quirks.
• LVM kills performance.
• Roll your own NFS locking. The standard one is insufficient.
• Use NFS hard mounts. Puts nodes to sleep until NFS server comes back online.
• RQ does not move data around. They use vsftpd to allow data to be moved.
• Constraints are good. Turns out many people and organizations operate under the same constraints.