Greg Luck's Blog

Jonathan Schwartz on open source

(Comments from the NetBeans session)

We have been held captive by the Java developers. The future is about reaching out to people without deep expertise.

The number of people passionate about open source licenses far outweighs those who have actually read them. Everyone is a lawyer now.

If I post a hit video to YouTube, someone else is monetising my free content. Something is wrong with that (Me: There is definitely a theme developing with all of these hints about compensating those who create software or content for free. So Sun is not only open sourcing all of their stuff, but paying others for their open source stuff. Wow).

What is Web 3.0 likely to be?
(my question)

Tim O’Reilly “Meaningless. Most transformative when we stop typing. 411 services Google, gestural interfaces. Each cell phone with an
intertial sensor. Instrumentation of people. Insurance based on GPS. Computers learn from us by what we do in our interactions
in the daily world.”

Tim Bray “Essentially unpredictable. e.g. Twitter. Don’t know but really exciting.”

Other Statments

Ian Murdock “A layer of abstraction is great until you really need to know what is going on underneath it”

Tim O’Reilly “Proprietary databases are the competitive advantage of the future. Critical mass of buyers and sellers made ebay a lock in. No one can touch it. All of the mapping
sites still use NavTeq.”

Tim O’Reilly “Google Competing to give you the result in line, rather than passing you off to another site. Google are starting to compete with their own users.”

Rich Green “Value of a platform is the ratio of the lines of code you write compared to the lines of code in the platform. Web 2.0 has that. What is the Web 2.75 Petstore?”

Tim O’Reilly “Web 1.5 was the Dot.com crash.”

Rich Green “Bill Joy. Where ever you work, most of the smart people are elsewhere”.

Tim O’Reilly “The architecture of participation”.

Rich Green “Scoop up the open source work of others and monetise it. Sun are looking at paying people.”

Tim O’Reilly “Horatio Alger said “Go west young man. You have to find something new.”

Tim O’Reilly “We are coming to the end of cheap outsourcing. Indian outsourcers seeing only 3-4 years of having a wage advantage.”

Tim O’Reilly “Users do their thing out of their own self-interest. “

Tim Bray “You can make money from the long tail but may be not in the long tail.” and Tim O’Reilly “Each long tail is attached to a very big dog”

The last few weeks I have been working on a single sign on/Kerberos project.
For anyone for whom either of those term is new, here is some food for thought. Windoze, Linux, Mac OS X (10.4), Firefox, IE, Apache, ssh… has, in the past 10 years been Kerberised. Rather than Microsoft’s Embrace Extend Annihilate being the death knell for Kerberos, their endorsement legitimised it.
Add in SPNEGO, another M$ innocation, which added browsers into single sign on, and here we are. Much of everything we need is Kerberised. An overnight success which took a decade. It is a very good time to adopt Kerberos.
Over the next little while I intend creating a Kerberos/Glassfish HOWTO, showing how to add a Glassfish security realm to a Kerberos realm. If you look in your JAVA_HOME/bin directory, you will see kinit and klist, Yes Java was Kerberised in 1.4.2. There is a Kerberos Login Config: com.sun.security.auth.module.Krb5LoginModule

Wotif.com, my employer, went live on Glassfish in mid January. I was the driving force behind the move. You can read about it on the Glassfish site here: http://blogs.sun.com/theaquarium/entry/how_to_run_a_top .
There is a questionnaire linked from the entry that is the raw question and answer source for the entry. You can see it here: http://blogs.sun.com/stories/resource/wotif/wotif_responses.html
Though not one of the main reasons for the change, we find ourselves on the JEE5 platform, and liking it. XDoclet boiler plate generation and EJB in-container testing are things of the past. It took a few days to move our code from EJB2 to EJB3. The end result is that we find ourselves early adopters of JEE5. The opportunitites this presents and some neat ways of capitalising on them are being documented on Robert Watkin’s blog. See http://twasink.net/blog/ .

Luke Welling of Hitwise showed up some trends using the Hitwise database. It is mostly site visits. Interesting to compare this with Google Trends and Tiobe.
Mailing lists seem to be getting replaced by feeds.
Ubuntu matching Debian in May 06. Redhat is still getting more visits than all of the others combined.
PostGres is gaining on MySQL and Oracle is in slow decline.
PHP continues its dominance over Perl and other dynamic languages.
Digg.com eclipsed slashdot.org in 2005.
Wikipedia is huge.
DEL.ICIO.US went crazy last year.
feedster.com dropped off after site difficulties and is now coming up.

Distributed Stuff

One was on distributed caching and file systems. We talked about memcached, ehcache and a new file system from Live Journal – Mogile FS. It is user space, so not really a true file system. It comes with a mod_mogile for Apache. Must take a look at that. It looks like use of memcached is settling on page caching. Serialization issues make it a bit expensive for object caching.

Emerging Internet Security Standards

This was a wide ranging discussion on whether any standards are emerging in Internet security after 10 years of nothing more than server SSL certs being taken up.
In Instant Messaging it looks like OTR, which stands for Off the Record, is emerging. Adium for Mac, which had a version 1 release today, has it built right in. Many others have plugins. You click a lock and it asks the other party to go off the record. If they have the software they can elect to. If not, the handshake does not complete and it stays in plain text.
OpenID is the other big one. It is a single sign on system for the web. Technorati and Wikipedia are moving to it. Firefox 3 will support it (Ben Goodger confirmed it is on the wish list). Firefox 3 is also down to support SAML, the Oasis standard. Sun’s Liberty Alliance specifies SAML. So hopefully something is going to happen here.
As to email, it doesn’t look like anything is going to happen. We speculated that it will take someone like Google Mail to introduce a standard. Maybe S/MIME, maybe something else. Interestingly lots of ISPs are supporting using SSL for MTA. I tested mine and it does not but apparently lots of Australian ones do. One tick for confidentiality.

Ruby Roundup

There wasn’t much on Ruby in the sessions. But I got around the rubyists. I met the guy who decided not to accept a patch for Oracle prepared statements. “Our user community does not really use Oracle and prepared statements can be harmful to PostGres performance and are neutral for MySQL”. One guy is an ex-Rubyist. His complaint was the community. Naive implementations solving problems that were elegantly solved 10 years ago, with the implementers thinking how cool they are. In the next breath I heard how banks and big enterprises are no longer interested in Java and are all doing Ruby. I look forward to them changing over to MySQL because the Ruby community does not care about Oracle.
There seems to be high respect for the JRuby project. It looks like it will imminently be able to run Rails. There are a few people showing how to deploy it on Glassfish. The Netbeans guys are making much of adding not just Ruby but Perl, Python and PHP language support. And IntelliJ IDEA released just a few days ago their Ruby plugin through their plugin manager. I played with it again (the last time was a few months back). It is ok. It has RDoc with CTRL-Q, syntax highlighting and minimal keyword auto completion. It needs to be much better to get me enthused. Allan Odgaard, the TextMate author, pointed out that Ruby autocompletion has just been added as a plugin to TextMate. (His focus is on supporting 100 languages, so he himself does not do too much that is language specific – he likes the Emacs analogy).
Tiobe announced that Ruby was the language of 2006, because it had the most growth. The latest Tiobe shows a leveling off of Ruby. Not sure if this is temporary or whether it will level off below Python.
Overall the super buzz around Ruby seems to be dying down.

Perl

Allison, a Parrot VM architect was here. I asked her about whether an open source JVM might get used by Perl and others. She thinks maybe, but the independent efforts, which are designed for dynamic languages will continue.

Segway

We all had a go on a Segway that the LiveJournal guy brought with him. It has a black key and a red key. Red is for fast. We all graduated from black to red and a had a great time. You delegate balancing to the machine. The rest is easy.

Google

The google guys were conspicuous by their t-shirts and tight-lippedness. Google’s share price has taken a bit of a tumble, which was the main discussion. Chris De Bono, helpfully “does not comment on future share prices”. This is very similar to his comment from OSCON two years ago in answer to questions about better Linux support with he “does not comment on future product announcements”. With conversation that exciting I moved on. Interestingly, given that there now a 1/googol chance of any new hires making anything out of Google employee optios, all the Googlers are emitting a new company line that Google has found new creative ways to entice developers to join. I am up to three recruitment approaches from Google now in the past two years and am probably now on the do not call list.
I did manage to meet Lars Rasmussen who was on the team that did Google Maps. Due to its Cuba policy the USA has lost Lars and his Cuban wife to a country (Australia) that has normalised relations with Cuba. A good win for Australia.

Java

Lots of Java people haunting the corridors. Thankfully they were all IntelliJ users, so we could escape the oppression of the Eclipse nazis for a short time. I met the guy who did the TestNG plugin. JUnit 4 is at the end of the day not that exciting. I might take a look at TestNG. I spread the word about Glassfish which I am now using at work in production. I think it will be big. It seems clear that Glassfish is the platform to be on for new experiments. Some of the non Java guys, who do not exactly keep up with Sun news, were interested to hear that Sun had done an open source Java app server. And the open sourcing of Java is big. Glassfish is coming on to the Linux distros (Ubuntu already announced) so they will see it.

Linux

It was agreed that Linux newbies prefer Ubuntu these days. A grizzled Debian veteran still prefers the original formula. I showed off my Macbook running FC6 on VMWare Fusion in answer to questions about whether I had used Parallels.

Politics

Lots of it at the conference. We even had the NZ Communications Minister with us. He seemed very well informed on IP issues. NZ is doing a new Copyright Act and trying to avoid the disgraceful Australian precedent done as part of the Australia – USA bilateral trade deal where we let the US export their IP laws to us in exchange for a little bit of market access for our farm products.
He suggested that patent infringement lacked mens rea. I suggested that it is now hard to write a line of code without infringing some patent. So we all have mens rea, but we have no idea what patent we are infringing.
Corporates still seem best served by building up patent war chests. A bit hard for those of who do open source.

I was at Enterprise Java Australia’s annual get together a few weeks ago.

The highlight for me was listening to Simon Phipps lucid telling of the open source strategy at Sun. It felt to me that the presentation we got was not too dissimilar to the one Phipps used at Sun.

It starts with Bill Joy’s observation that “You can’t hire all the smart people”. The solution to this very real problem is that open source gives you access to them. You don’t need to hire the smart people if you can access their output and you are not trying to compete on that.

Given that version numbers are now popular Phipp’s has his own: Software Market 3.0. (Didn’t anyone ever tell these guys you never use a .0 version of anything?). In this market you pay for services and software at the time of production, not before.

The mantra for all of this:

1. Collaborate over what does not differentiate. Do this using open source. Encourage open standards, because these mean substitutability.
2. Compete by innovating on top of the commodity base.
3. Contribute!

So how is this being applied?

Sun Application Server 9 is open source => Glassfish. Sun’s Message Queue 4 is open source => Open Message Queue. The JDK is going to start to be, in November, according to Phipps.

And what are the effects?

Application Servers

For many years I have been using Orion as the standard server I install on new products. It was cheapware and I could get it in without a lot of fuss. Last year I switched to JBoss. It was OSS, and worked well in the applications I used it in.

Right now I am moving from Orion to Sun Application Server. The reasons are:

1. JBoss’s Marc Fleury just gets up the nose of people. Some people hate JBoss and will not use it.
2. I don’t want an expensive app server, because I will spend all of my time justifying new licenses rather than developing. Getting it approved for some applications and not others just leaves a mess.
3. Sun App 9 is just that: version 8. The biggest problem I have had with it is the quick start feature where services are started lazily, which is new in 9 – it is a great idea but buggy. Turn it off and things work great.
4. Glassfish is open source. You can step through the App Server code. If you get a bug you can often see the fix for it and add that to your bug report. The worst case is that you can patch your own version temporarily.
5. Glassfish is one of the first JEE 5 implementations. I will move to a POJO architecture by moving to EJB3. But according to Ben Alex, both EJB3 and Spring are POJO architectures. Josh Mackenzie of POJO fame, Paul Hammant and the light weight crowd have won. But it does not mean you need to use Spring to achieve it. It looks like migrating to EJB3 is going to be fast and painless. And no XML. Nada.
6. I can buy production support and training. Prod support is about AUD1500 per year per server and the training is about AUD800. Cheap. But revenue Sun was not getting from me before.

For Sun rather than getting a large licensing revenue from a small number of customers, they potentially get a small amount from a large number. They will be getting money from me they never got before.

In I, Analyst the blog of Richard Monson-Haefal Richard points out that Sun’s market share grew by 6 percentage points over 2005 to 19.7% as of December 2005. See http://rmh.blogs.com/weblog/2006/05/bz_research_on_.html . He credits Glassfish as the likely reason. It will be interesting to see how things look this December, as Glassfish was only released in May.

Open Source Projects
We are currently using ActiveMQ 3 series. We have hit some bugs with it that seem to be longstanding. The 3 series is now being somewhat maintained by the Gluecode guys. With Glassfish we are using the built-in Sun Message Queue, aka Open Message Queue. Once again this is a version 4 product. It is being maintained by Sun Engineers. All of the enterprise features, including clustering, are there. Does this mean we will upgrade to ActiveMQ 4, or does it mean we will move to Open Message Queue. Time will tell.

I am a little worried, as an open source maintainer myself, that Sun’s dumping of commercial quality software onto the open source world may kill many open source projects. But perhaps we will all finally have achieved our aim if we our own projects become obsolete.

Language Implementers
During an extended language discussion with Dave Thomas (not the prag one) a few months ago he would keep dismissing languages as toys. Finally it came down to a language is a toy if it does not have a decent virtual machine/runtime interpreter. The only two that are decent: JVM and CLR. I think Dave might have a point here.

Some stuff going on:

• Perl is creating a Parrot virtual machine

• The Ruby world has one right now that is 15 times slower than C. (see earlier blog) There are lots of VM projects. See http://wiki.rubygarden.org/ruby/page/show/VirtualMachineOptions. Ruby2 is planned to use Rite. See http://wiki.rubygarden.org/Ruby/page/show/Rite
• Microsoft needed a multi-language VM from the start. Projects to run Python (Iron Python), Ruby (Garden’s Point Ruby.Net – see http://www.plas.fit.qut.edu.au/rubynet/) are well under along.

Clearly every language needs a great and performant VM that runs on all architectures and OSs. Why should each language implement its own? The answer is trust. Would you trust Microsoft? Or Sun? But if Sun open sources the JDK, that solves the trust issue.

Sun are adding invokeDynamic and some other goodies to the next JDK to better support dynamic languages. And they are open sourcing it. If this is done right, the other language implementers may drop VM work and just provide a JDK implementation. This strategy could see the JDK (renamed to some cool name of course like my suggestion of Hydra Virtual Machine) move into a central position in the open source world that it has never occupied before.

Couple this with Sun’s new USD99 per incident developer support offering, and once again, Sun may see revenue they have never seen before. Once again of the type small amount * large number.

The law of large numbers

In summary, I think the totality of what Sun is doing amounts to a revolution. It may touch the lives of many of us. If it does the law of large numbers may see Sun’s coffers brimming over with software and services revenue. I am not sure if this is Jonathan Schwartz’s doing, but someone at Sun seems finally to have come up with a great software strategy.

New trends:
Ruby books are now outselling Python and Perl books. But JavaScript books have increased the most and are outselling all of the other dynamic languages. Why? Tim thinks it is driven by interest in Ajax, which is the hottest thing right now. Time to get over my JavaScript hatred. IntelliJ helps with that a lot. (Also interesting that Rhino is bundled in JDK 1.6)
The other thing new on the horizon is Django. Tim pointed out that both Rails and django grew out of closed source projects: Rails from 37 Signals, and django from Lawrence Journal World. django is Python’s answer to Rails, so it will be an interesting one to watch.
Virtualization is a big new trend.
Another one is that being on someone’s platform increasingly means you will also be hosted on their infrastructure.
Open Data. Owning your own data and being able to take it with you.
Firefox is the equivalent in the browser world. (Tim did not say this but someone said something yesterday about Eclipse being the new Emacs).
Asterisk and open VoIP is a big deal.
Ubuntu is on the rise. It has huge interest relative to RedHat.

(Update: Wow I got a record number of comments to this blog. Answers to some common themes at the end of the post)
Who are those who are benefiting from Ruby on Rails? Answer: O’Reilly Publishing, the authors Bruce Tate and Dave Thomas and a handful of consultants.
At last year’s conference, Tim O’Reilly had carefully analysed his book sales and was desperate to identify the next big thing. Same for the pragmatic programmers and for consulting companies wishing to push the next big thing. C# had been and is a disappointment. Despite a huge push it refuses to move up the Tiobe programming index.
I get the feeling that everyone needs a next big thing, and if there is not one, they create it. So what has happened to Java, after the spate of Beyond Java books? Answer: according to tiobe it has risen higher. So what is declining? VB.net and Perl. Not Java.
So am I ignorant of Ruby on Rails? We have two production applications running on Ruby. And how is it. Well, despite being perhaps no more than 5% of the functionality of our applications, Ruby on Rails is the number one consumer of Oracle CPU and logical gets. Why? Rails does not support prepared statements, so Oracle has to reparse every time. This is something that Java has had for years and years. And ActiveRecord seems not to have learnt Hibernate’s lession; that OR tools suck for performance and need caching tricks to make them work well. Also, our Rails apps running in (now unmaintained) fast-cgi regularly go awry and fork more processes. Each one creates a new connection to Oracle. So, the opposite of connection pooling; connection denial of service. And does Ruby support Unicode. Not really. And is Rails threadsafe? No. So, is it Enterprise Ready. Absolutely, according to those with a clear vested interest in supporting the next big thing.
Are these problems solvable? Yes, in a fashion. For example, I have been told that lighthttpd and Mongrel should be used rather than fast-cgi. And MySQL should be used rather than Oracle.
And does it matter that Ruby is 15 times slower than Java. Of course not. How could it! Just buy more hardware. And more hosting costs. And more System Administrator salaries.
After all, the productivity benefits of Ruby are so much greater than Java you will save all of the money in development. Or do you. Our experience was that Ruby on Rails took longer than Java would have. And what about maintenance. Well we just refactor as things change. Or do we? There are no Ruby tools that support refactoring. And nor are they are expected due to the difficulties of implementing refactoring tools for Dynamic Languages, or so I am advised.
And what about support. Well there is the Ruby mailing list. Which is quite active. That should be good enough for anyone.
And what about Python? Python is arguably a more mature dynamic language with a much larger developer community and number of libraries. Why does it suck? Underscores and “self”, according to one of the leading Ruby advocated. Wow, that sounds really bad. Doesn’t Martin Fowler’s Refactoring book recommend _ for fields? Must be deprecated. Oh and what about Django? Now this really sets the Rails people off. Why? Because it is Rails like. A rifpoff they say. Yes, but where do the Rails ideas come from. From my point of view it exactly what I have been doing in Java for years and years. So if Ruby can rip off from Java (most welcome BTW) why cannot Django ripoff Ruby on Rails? Answer, because the vested interests have decided there is money to be made from promoting Ruby, not Python.
In short, anyone who questions the benefits of Ruby on Rails is not with the program. You know, once upon a time, being open source meant being better because of improvements spurred by constructive criticism. Its about time the Ruby on Rails community accepted some.
Anyone for Haskell? Or J2EE 5?
(Answers to common comments:
1. Where are the line breaks?
Ah, sorry for that. This was a late night post, after a drinking session. Forgot to check “Convert Line Breaks” in the Text Formatting option. Also, I have been using Writely lately to blog, until it broke
2. Thanks for saying what everyone is afraid to?
No problem.
3. Why are you so ignorant?
I am capable of learning. Really. Enlighten me.
4. Why are there no references to back up your claims?
Some of the claims are explored more fully in other blog posts. The one about Ruby performance is based on freely available benchmarks. How important is it? The database is normally the slowest part, but it depends on your app.
5. Who was the nasty Railroader who claims django is a rip off?
I personally did not realise that django predated Rails. The Railroader was very tetchy about any success django might be having. He described django as “Rails-like”. The rip off part I think was was a false connection I made. I do however think that the django crowd have realised the importance of hype. So perhaps to that extent they have learnt something from the Rails crowd.
6. What about Haskell?
Sadly no one that posted was that interested in Haskell. Our speaker yesterday said it takes a year to get into it, so maybe that is why.

I do not normally take to buzz words, hype and shiny new toys. These things usually annoy me.

I do however think there is something to Web 2.0. I don’t want to go into what is meant by that here. See Tim O’Reilly’s landmark essay for a definition. If you look at the technologies that go into it, most of them are late 90’s. While Tim provides a broad explanation of what Web 2.0 is, for my purposes I am interested in the following technologies:

• CSS, semantically valid XHTML markup, and Microformats
• Unobtrusive rich application techniques (such as Ajax)
• XUL
• SVG
• Flash Remoting
• Syndication of data in RSS/Atom
• Aggregation of RSS/Atom data
• Clean and meaningful URLs
• Weblog publishing
• REST or XML Webservice APIs
• Some social networking aspects

Some of the browser standards introduced in the late 90s are:

• W3C Document Object Model (DOM) Level-1 Specification (October 1998)
  

• W3C RDF Recommendation (February 22, 1999)

• W3C CSS1 Recommendation (December 1996)

• W3C HTML 4.0 Specification (April 24, 1998)

So why are we proclaiming Web 2.0 in 2005/6? The answer I think is two fold:

First, I feared but expected that with IE dominance, Microsoft would go on to define a Web 2.0 of its own; one that work with Windows. While they have played a major part in Web Services it is the community itself that created the new platform – just like Web 1.0. The Web 2.0 stack prefers open standards and standards based browsers. Firefox and Safari both led the charge with built-in RSS readers and concern for web standards. Microsoft’s dismantling of their IE team led to a period of stability where it was possible to build frameworks by working around limitations. Microsoft, very strangely in my experience, is paying some attention to web standards. A review of the IE blogs shows that there will be some movement toward greater CSS compliance, together with RSS support. IE7 is pretty much a catchup to Firefox/Safari for Microsoft.

Second, there is the network effect. Metcalfe’s law states that the total value of a good or service that possesses a network effect is roughly proportional to the square of the number of customers already owning that good or using that service. A mobile phone example of the power of this law can be found in SMS. All GSM phones have SMS. While the 3G bandwidth speculators bid up spectrum licenses to dizzy heights the Generation Xers and Ys stuck to texting. SMS was created in GSM Doc 28/85 rev2, June 1985. SMS was ubiquitous precisely because the standard was so old. New standards such as MMS are not supported on all phones. People stick to what they know will work.

So, standards are introduced in the late 90s; time ticks by; an installed base grows and becomes ubiquitous; some smart developers build to it. Voila! Web 2.0. Tim’s article partly defines Web 2.0 through exemplars. I think one of the best ways of web2ifying your mind is to configure netvibes as your home page. Netvibes is a great example of Web 2.0 in itself and lets you add portlets? widgets? Web 2.0 bits? from plenty of others.

I think what comes next is competitive advantage by being Web 2.0 compliant. A bit like when the web itself was new, and you were either on it or not. Back then there was a competitive advantage in going online. I think there is a window of opportunity right now to gain competitive advantage from Web 2.0.

(Attribution: Many of the links for this post were shamelessly lifted from Wikipedia)