Open the source, sesame

About a month ago a big shot from Microsoft came to speak at the university - looks like he's on some kind of a tour, and his mission is to convince universities to teach Windows operating system aside Unix/Linux. He waved at the audience with a free license and tried very hard to prove that some aspects of Windows design are better than Unix, mainly because it is "newer" design and better suited for nowadays computers.

Long story short, one of the things he said, was that open source is bad, because engineer who looks at the source of a library will design his application assuming particular implementation, which is now tightly coupled to library internals - bad. Obviously a discussion erupted - design by contract and Eiffel were thrown in the air, relation between programming language and operating systems..., but frankly all I could think of is my distant past as a VB developer and those 1000 pages "Windows *** Unleashed" books. They were written by hackers who used trial-and-error against Windows DLLs trying to make some sense of the APIs - the formal documentation was either sloppy or intentionally incorrect to misguide us, Windows application creators outside Microsoft, since we were potential competitors. Things were so different when I moved to Java. (Though my first Swing experience made me - believe it or not - miss VB, but that's another story.)

Anyway, statement like this coming out of the mouth of a Microsoft employee, even if he has an impressive Unix record, was easy for me to dismiss. But here is a respected Smalltalker saying

"I think people should have the source ...not to get miss quoted saying I'm against open source, but I think it's important class libraries should be viewed like caves."

(Ah, wait, he is from IBM - he can be ignored too... just kidding)

I mean, seriously look at this puzzler for example - it's pretty cool, but the solution heavily relies on a particular implementation, I would even say this is actually a hack. And sometimes a hack is necessary... or is it? Is there or will there ever be a perfect environment where we can program without hacking?

P.S. The reference to Ali Baba tale in the title is intentional. Open source is good, very good, it's a treasure and in the world dominated by large corporations - almost a miracle. But IMO we should resist the temptation to exploit it to the last bit, because we might find ourselves locked in the cave.

Static methods as function objects

In one of the earlier posts I described a policy object created by "functional style programming" in Java. The basic idea is to make methods into Function and Predicate objects and then composite the logic from these building blocks. The problem though was that the policy implementation looked alien to Java. To re-cap: I wanted to invoke a different function based on the class of the object - classical multi-dispatch with a single argument.

So inspired by extension methods idea I added a new mechanism to define properties (and functions in general) - via static methods. You define a bunch of static methods with the same name (or appropriately annotated) like this:

static int size(Collection c) {
 return c.size();
}
static int size(Map map) {
 return map.size();
}
static int size(Object obj) {
 return (obj.getClass().isArray() ?
   Array.getLength(obj) : 1);
}

Now assuming I have a property declaration

Property<Object,Integer> size =
 new Property<Object,Integer>(0) {};

The underlying implementation of the size.of(object) would be a "policy" (composite function) dispatching to one of the static methods according to run-time type of the object. So that size.of("foo") is 1, size.of(Arrays.asList("a","b","c")) is 3 and size.of(null) is 0.

There are couple of issues though.

Issue #1: automatically ordering the rules. Now when rules are not added manually, I should be careful to test for more specific class first. The solution is to calculate a distance between two classes and always look for a best match - basically same algorithm javac uses when choosing methods at compile-time. This isn't fully unambiguous though: if class C implements 2 interfaces A and B, having different implementations of size(A) and size(B) would put me into a dilemma which one to choose for size(C). Javac in such a case reports an error - thanks to Java being statically typed. But once I make it a run-time decision the only logical solution is to report an error at run-time. To make the problem less acute, I could add an annotation that controls ordering of the methods.

Issue #2: this is less generic than a general predicate-based policy - it is limited to "instanceof" predicate. One of the policies I use internally in the properties project is reflectively invoke getFoo() if the class has a "getFoo" method. This isn't possible with static methods.

Issue #3: reflection. It's not really a problem, but rather an enhancement I want to make: instead of using reflection I could generate the functions at compile-time with APT, like Bruce Chapman does here. This is one of my next adventures, which I will hopefully describe in a future post.

Digg a pony

Here is another post with no helpful information, sorry, I meant to write a meaningful piece but I am on vacation and in fun spirits :-)

Anyway, I just got reminded of something that crossed my mind before and is now more relevant than ever - the term John Lennon invented almost 40 years ago: "bagism". It's amazing that at the time everybody laughed - a person giving an interview in a bag. You can look this up, or even better watch the video (fast-forward about 2 and a half minutes). Here's a short re-cap:

"if everybody had to go in a bag for a job (interview) there would be no prejudice, you see, you'd have to judge people on their quality within, you know, we call it total communication..."

I think this is really the web: blogging, mash-ups, syndications, and social networks, and why all this is a lot of fun - you really know nothing about the one you are talking to, except what he (she? it?) tells you.

Dig a pony lyrics mention

"penetrate any place you go" (...internet?...)
"radiate everything you are" (...wireless?...)
"imitate everyone you know" (...blogging? syndication?...)
"indicate everything you see" (...search? google?...)
"syndicate any boat you row" (...social networks?...)

Cheers to total communication!

P.S. A tribute to Passover holiday (the reason for my vacation): there were once four boys (ארבעה בנים) - the wise and wicked one, the cute one, the quite one, and the one that can't sing... and they were fab.

Java and closures

My reaction to the likely postponing of closures till Java 8.

Mercurial adventure on Widows

With a friend at work we decided to check out Mercurial as source control system. The repository was set up on a Linux box and zillion lines of code and libraries were sucked in from ClearCase, now I just had to clone it to my lap-top running Windows XP.

Installed Mercurial 0.9.5 via TortoiseHg (I am familiar with Tortoise with SVN so it was a natural choice) - very easy and straightforward so far. Issued clone command - hm, filename too long. Switching to cygwin - doesn't help.

So Ok, we have pretty deep directories structure and Mercurial makes it worse because it translates everything to lower case by "escaping" capital letters - adding underscore before the letter, so "A" becomes "_a" and a single underscore becomes a double one. Why? This is to avoid problems when working with case-insensitive operating systems like Windows. Fair enough, but what do I do with these long filenames now... Python bug? Oh, great. In the meantime the error became "file not found", but I (correctly) suspected the cause is the same.

I found this bug report (and this one) - at least it happenned to other people, it says I can use a 'usestore' workaround. Cool, found this - says change one of the scripts, but hey, they are compiled?! Ok, downloaded ActivePython (I admit, I never touched Python in my life) also downloaded the sources. But apparently I need a C compiler too for the full build from source - that's too much, I decided to just run Mercurial with the interpreter.

Found the right script, changed it. Boom, of course extra space in indentation - until now I only heard jokes about this (besides similar experience with Fortran half a life ago). Nevermind, got it now. Still not working. I take another look at the script - the if statement I just added was already there before! Oops, what's going on? Trying to understand the Python code - it is looking for some 'usestore' configuration option, which BTW is supposed to disable the "escaping". All I need now is to set it. Hm, how? Eventually I find out that I need to edit <TortoiseHg-install-dir>/Mercurial.ini adding

[format]
usestore=False

(If you use plain Mercurial, not Tortoise, create Mercurial.ini under <mercurial-install-dir>)

I do it and... voilà, the repository cloned!!! So bottom line - all I needed was ini file change. It is somewhat dangerous 'coz I still have the mixed case, but at least I have the code on my laptop. Why didn't I find it in an FAQ or tips'n'tricks or something? No idea. Hope this post helps the next person who googles for it.

Blonde Moment

Kirsten Dunst wearing Red and Sensual Java.

The taste of Java Generics

As the closure debate flames are rising, people are attacking Generics in order to attack closures. Some use it as a negative example against particular closure proposal, some try to discourage Java language changes in general and some to promote other languages. When I read articles that start with "Generics are hard" and lead into "Generics are bad", I can't help thinking of the fox and the grapes. Yeah, they are hard. So is concurrency, NIO and many other things. So? That's why people create frameworks on top of Java.

Don't get me wrong - Java Generics have faults, and they are long known. Here's a hit-parade of my biggest complaints, but I will also mention some solutions I have used:
1. Erasure
The way to work around it is by using type token or super-type token. As any workaround, it is limited, but it can solve part of the problem - Class#cast and Class#newInstance don't issue the annoying warnings.
2. Generic parameters not covariant
It is usually a good idea to use bound wildcards extensively.
3. No lower bounds support (except wildcards)
We can use static methods instead (ala extensions) with <T extends Super> rather than <S super Sub>
4. Verbosity, instead of more aggressive type inference
Not much to do, hm, learn to type faster? :-)
5. The angle brackets
Yeah, I hate them most when typing on blogger, luckily I am old enough to have been coding HTML in plain text editor a long time ago, so... back to GT and LT escapes.

BTW Java didn't die, it has simply grown beyond its hipness age. And I am glad that it spawned even better things, such as Scala. But there is a difference between "good" and "popular". Java is popular and not that bad, other languages may be better, but not as popular. Will Scala be willing to sacrifice some of its expressive power and shortcuts (gee, stuff like that reminds me why I don't code in Perl) to become simpler to learn/read and potentially more popular? Should it? (Ruby consciously didn't and ... didn't.) We shall see.

Listen To What The Man Said

Just couple of nice quotes for today: Alan Kay on Innovation and the not so new but still relevant Edsger W. Dijkstra transcriptions - look at this Q&A for a quick feel.

"Machine capacities now give us room galore for making a mess of it. Opportunities unlimited for fouling things up! Developing the austere intellectual discipline of keeping things sufficiently simple is in this environment a formidable challenge, both technically and educationally."

Wildcards observation

"A bug in poker is a limited form of wild card."

- Wikipedia

While looking for academic references for my properties project, I bumped into the expression problem - the write-up by Phil Wadler and follow-ups by Mads Torgersen and Matthias Zenger/Martin Odersky are really fun and educative readings.

This stuff caused me to re-read the the wildcards paper and I suddenly understood why I had this strange gut feeling about wildcard usage. Almost all learning material about Generics, including the wildcard paper itself, emphasizes the distinction between read and write methods - having a wildcard in a collection parameter prevents you from adding elements to the collection, but allows reading. This is true, but somewhat misleading. Actually, wildcard in the parameter allows us to call a method that returns the thing behind the wildcard and treat it as if the returned object is of the type of wildcard bound (or an Object, if the wildcard is unbound); the wildcard prevents us, however, from calling any method that takes "the thing behind the wildcard" as a parameter. Now obviously, to add something to a collection you need to call a method that takes that something as parameter, and it is indeed prevented, but other modifications are not disallowed, e.g. clear() .

The consequence of wildcard usage applies to any method, not just "write method", for example - containment test. So looking at Kevin's example, it now seems to me that doSomeReading method shouldn't have used a wildcard if it wished to perform a safe containment test - that's the catch (or should I say, hm, capture...)

Happy New Year!

Past forward

Since my previous post was devoted to Java history and this interview with Martin Odersky recently came out, I decided to do one more short transcription related to Generics history.

"I am an academic, actually, I'm working at the university in Switzerland. ... A long time ago I was a Ph.D. student of Nicholas Wirth, so I am in the Modula 2 operand camp... But then I drifted more and more into functional programming community, I have a good friend named Phil Wadler who is very active in the community, so he sometimes told me ... "there is this new thing, it's gonna bury functional programming, it is called Java". I wanted to find out more and he said it has garbage collection, and it runs everywhere on the Web ..., so what are you gonna do against that?!

So ... we said we'll try to do a functional language on the Java platform, because the Java platform looked very exciting to us then (it was in 1995, so a long time ago). ... To find out more about what it is (I always learn best when I write a compiler) let's write a Java compiler. We had this second ever Java compiler out there after the old JavaC.

Then we said - if we want to do a language it should be Java compatible, and we wrote a language called Pizza, which added stuff from functional programming to Java. We experimented a little bit with that and then Sun came and said - we like what you did, in particular the Generics, we wanna do that. Then we had a project with Gilad Bracha at Sun and we did this GJ thing, which was the Generics thing.

The compiler for GJ eventually became the current JavaC compiler, Sun took that over because they decided that this compiler was more maintainable than the original. (Interviewer: "from JavaC 5 on...") From Java 1.3 already.

So the compiler was in 1.3, but the Generics were disabled. In fact, the Generics were at first not that much disabled, there was a secret switch that people found out about, so there was this stealthy movement of people using Generics already in 1998 when it first came out. And then Sun pulled the plug, they said - you can't do that, so you have to re-write your compiler so that it doesn't make any Generics available. So there was a special mutilation script in the compiler which would rip it off, so that people couldn't use it for a while, until it came back in Java 5."

Back to the future

If you look at the programming language history poster there's a bunch or arrows leading to Java genesis, one of which comes from Smalltalk. Ok, I heard about Smalltalk and saw some code in books. I also knew that Gilad Bracha was one of the authors of Strongtalk (Smalltalk with static types) and later immensely contributed to Java evolution while working at Sun. I also saw references to this supposedly great language called Self in some programming language articles. But only now I see how it all linked together thanks to Avi Bryant's Keynote at RailsConf (yeah, of all places to learn about Java...) so I decided to transcribe here a small portion of this fascinating talk.

"There's a legend that there's something inherently about Ruby that makes it hard to implement a fast VM for it, because of open classes, because of dynamic typing - there's something about meta-programming, something about Ruby that makes it hard in this day to make it run fast. And it's just not true! It was true 20 years ago - it was a hard problem, but people solved it.

This is actually from Sun site, this is a bunch of papers that collectively explain how do you make Ruby go fast. But you'll note that they're written in 1989-91, you can go and look at the URL here, they come from the Self project. And the history is kind of interesting here, so since this is about the future and about the past, I'll give you a really brief history.

Self was a project at Sun to do Smalltalk, but even more so. Even purer object-oriented. And it meant that they had to find ways, because they wanted to make it so object-oriented, so pure, so turtles all the way down, they had to find more and more ways to make it go fast. And the technology they came up with was so interesting that they actually spawn off a start-up to try to implement new super-fast Smalltalk that was gonna be used on Wall Street. But before they got to release it, this was gonna be called Strongtalk, before they got to release it, Sun bought them back.

And Sun used the technology, used the HotSpot profiling and Just-In-Time compilation technology that have been developed here to build the Java VM. Which is sort of one of the great tragedies of technology, but it's true, that Java HotSpot VM in fact is what came of all of this great work of making dynamic languages go fast - of course they crippled it so that it doesn't make dynamic languages go fast anymore, but maybe that'll change. One interesting footnote to this is that a lot of the engineers that were involved in that have since left Sun and are back to doing Smalltalk."

Extension methods proposal

I'd like to talk here about another Java 7 feature proposal lobbied by Google that generated lots of waves in the blog-sphere after being revealed by Neal Gafter. A lot have been said (Peter, Alex, Stephen, Rémi and Stefan) so I'll add just 2 humble observations.

Adding capabilities to the language vs. maintaining a safe and clear programming environment is a tough balance to keep. I had my doubts in this particular case, and I think I tend to favor it, with one modification. I think it would be better if extension method import was different from normal static import, so the developer knows exactly what he is doing. I propose following syntax (that BTW does not require any additional keywords in Java)

import static java.util.Collections.sort
  extends java.util.List;

Compiler would then check that the first argument of sort is a List (or its super-class or super-interface) and compile

List list = ...;
list.sort();

into

Collections.sort(list);

The main advantage of this proposal is that extension methods are imported consciously, rather than implicitly.

The "importer" is also given some control over the overloading rules for the extension method - what if Collection has a sort method? what if ArrayList does? well, with the syntax I propose here the developer targets the method to a specific class, so overloading rules can be applied just as if the method was in that class, in this example it would overload Collection one, but not ArrayList one. It reminds me of the Ruby feature that Neal Ford had mentioned - calling a method, and if it does not exist, executing some code piece instead. So with proper overloading rules, could imported extension method be Java alternative for method missing? At last, there's this dark corner question "What if List had a sort method?" Personally, I think that if the class explicitly imports an extension method, it should be treated as local definition, while the method defined in List as inherited definition, and therefore (following the principles laid down by Gilad Bracha in one of his recent papers) the extension method should override the one in List. This is somewhat similar to the infamous "Mother May I" rule, but I better leave this for greater minds to solve.

Another interesting aspect of extension methods in general is their behavior in case the first parameter of the method is generic. Take Collections.max method for example. It accepts collections which element type extends <T extends Object & Comparable<? super T>>. AFAIK there is no definition of extension method behavior with generics. Maybe they are not allowed, but if they were, I would expect that list.max() compiles only if elements of my list are compatible with the static method definition, which in this particular case means that they should be Comparable to each other. Now this is an enhancement of Java generics system - being able to add methods to classes based on their generic parameters! Wow, it's somewhat spooky, but really powerful, I can imagine a few neat things I could do with this :-) and since I am now going to spend some time dreaming, I am going to leave you to think about this...

P.S. (12/12/07) Just came across a vivid discussion of similar feature in .NET. First reaction - "ah, oh, I get it now, it's another one of those .NET has it so we gotta have it too features!". But seriously now - let's learn from their experience, the conclusions are pretty similar to what is discussed here.