Eclipse Ganymede is out

.. and one hungry Indian dude wrote a section in the New and Noteworthy.

Immutable locks

Immutability is something that's mentioned over and over when it comes to parallel programming. Part I of Java Concurrency In Practice is all about composing objects that "play well" in the concurrent world and those that are immutable are the "best" citizens. No wonder functional languages are coming up in a big way. Anyone want to bet on them taking over the world in the next decade? Or if not taking over outright, at least succeeding in mutating our beloved imperative ones into unrecognisable functional beasts.

Anyway, check out the method below:

public class Foo {
    private Listener[] listeners;

    public Foo() {
        listeners = new Listener[0];
    }

    public void addListener(Listener listener) {
        synchronized (listeners) {
            Listener[] newListeners = new Listener[listeners.length + 1];

            for (int i = 0; i < listeners.length; ++i) {
                newListeners[i] = listeners[i];
            }

            newListeners[listeners.length] = listener;

            listeners = newListeners;
        }
    }
}

So here, the first thread to obtain the lock on listeners reassigns it to a new Array object... newListeners. Subsequent threads would continue to lock on the old listeners array, while new threads would lock on the "new" listeners array, and potentially corrupt the data. So I guess there's an unwritten property about locks... they need to be immutable to avoid situations like the above.

So given that locks need to be constant, there's no way to have a workable solution in the above code without using an additional object as the lock. The easy "lazy" thing to do is to just synchronize the method itself. The Foo instance (this) would then be that "additional object". But that would be wasteful since it would prevent all other synchronized method calls, even ones that have nothing to do with listeners.

(If instead of an Array, a List was used, the problem wouldn't have occurred given that the List itself wouldn't change. And I think, nowadays mostly the thinking when programming in managed languages is to go for Collections, rather than mess with manually managing Arrays. But I have still seen code like the above, so it's not totally contrived.)

Constant Learning with Miro

In every field, it's important to continue to learn about one's discipline even after graduating from formal learning institutions. No where is this more pertinent than in the software industry where the only constant is (fast and furious) change. There's always books and articles to keep up with what's new, but videos have emerged in a big way as another avenue.

And it seems like the developer communities at all the big guns... Microsoft, Sun, Google and Yahoo... have jumped on the bandwagon and are producing great content. Microsoft had started channel 9 a few years back where they'd go around interviewing key engineers. Sun has videos through the Sun Developer Network Channel. Google has their Tech Talks series. Rahul introduced me to Yahoo's YUI Theater few months back.

The great thing about a lot of the talks is that although they are presented in the context of the company's platform/language/technology, they tend to transcend them and are concepts and trends that apply to general software engineering. Microsoft's channel9 for instance, has had wonderful discussions with architects and designers on everything from functional programming to garbage collection to concurrency.

In spite of all the great videos out there, it's a hassle to have to go to each of these different sites, see what's new and watch it. Enter Miro. Miro is an amazing open source application designed specifically to consume vidcasts (it works with podcasts too). Here's a screenshot where I've subscribed to some of these "channels" (yes, I have a lot of catching up to do!).



You may be wondering if this really is a huge deal. I think it is. Miro simplifies and automates the process and makes it so simple and easy. Every time you start miro, it'll tell you of the latest content available and ask if you want to download it. And the quality is generally much better what with being able to view it full screen and it also has nifty features like remembering where you left of in case you have to pause midway through etc... The one downside is that it has to download the video bits as opposed to streaming it through flash (as is common on many sites since the emergence of youtube). So for the bandwidth constrained it can be a bit of an issue. But then again, flash still has some issues on Linux and some of the videos don't play. Plus, looks like Microsoft recently converted to Silverlight which doesn't work on Linux (haven't looked into Moonlight). So most likely you'd end up having to download it anyway.

I've only mentioned few channels, but as you can imagine, there's tons of them around the web. Miro has a guide built-in which showcases some of them. Apart from tech content, many universities like Berkeley and Princeton are broadcasting some of their lectures and events. So there's definitely no lack of content. Just need to make time!

Overall, I love Miro and can't say enough good things about it. Kudos to the guys who've developed it.

India Web 2.0

Came across this news site tracking India's contributions to Web 2.0.

avashya.com

Re: Growing Pains

I guess it started with the .Net guys getting aggressive with new features like annotations, generics, delegates and now Linq. They are becoming more dynamic. Something to watch out for.

I think the fact that .Net is a closed source platform controlled by Microsoft is a huge factor in them being able to be so aggressive. It seems like today, when everything is moving towards getting open sourced, this should be seen as a huge red flag. And in some sense I feel that way, but I mean you really have to marvel at the stuff they're doing. When you read the Evolution of LINQ you see some really neat concepts like Extension Methods and how they've used a previous language feature (Attributes) to actually implement it.

Regarding going the dynamic direction. I'm not sure that's necessarily true in the "core language/platform philosophy" sense. Both .Net and Java were designed from the ground up to be strong typed and will stay that way. I think it's more about abstracting out some tedious typing when it's possible for the compiler to infer the type... i.e. it's just syntactic sugar. It's especially useful when dealing with generics (and the endless <>s). So it's still going to all be the same under the covers, unlike in dynamic languages where there is no type safety at all. (Again it's interesting to see how var in C# came about as a necessity for another feature... Annonymous Types... which were needed for LINQ).

Java had to respond and got annotations and generics. Generics has been a mess with wildcards. Closures seems to be going down that route. If only we get something thats simpler and yet extensible. The memory model is simpler and concurrency API has also been a great addition.

Lets see how long before Java steals LINQ and related features ;) They're still arguing about closures and how the generics implementation was overly complicated (ppt). Wonder how long the insane backward compatibility requirement will continue. Also when Java was open sourced last year there was general optimism that there would be a lot more innovations to the language. But has it just cause a lot more arguments and disagreements about the direction? Java 5 was big release, but since then there hasn't been anything major.

My 2 cents for the year.

Growing Pains

These are pretty exciting times. A lot of languages are being extended to add more features...

I guess it started with the .Net guys getting aggressive with new features like annotations, generics, delegates and now Linq. They are becoming more dynamic. Something to watch out for.

Java had to respond and got annotations and generics. Generics has been a mess with wildcards. Closures seems to be going down that route. If only we get something thats simpler and yet extensible. The memory model is simpler and concurrency API has also been a great addition.

C++ seems be following on Java's line with a similar memory model. I dont know what else is there but C++Ox should be big.

There seem to be a lot of changes coming into Javascript and they want to make it much more Java like. Just heard a talk on the proposed changes and it seems much more complicated (read crappy). There seem to be so many new keywords that it just does not make sense.

Is it time to move to a language like Scala? Cause I've not been able to leave static checking for Ruby yet. Hey why not just listen to Paul Graham and code in Lisp :)

Profiling - Just works

I've been trying to set up a profiler for Java development.

I use Eclipse primarily and my first choice was the Eclipse project TPTP. However the damn thing just refused to work on my machine(Windows 32). I needed to primarily attach to externally launched Java programs. The Agent Controller (which collects profiling data) sample scripts works. But Eclipse is still not able to find the Java process. And the same installation/setup steps worked on another machine. But memory profiling did not work there!

That's when I tried the latest RC build of Netbeans. They bundle their Profiler with the IDE so no need to install anything separately. They have a pretty nice UI wizard which gives directions to start the java application for profiling. And guess what.. it Just Works. Though some of their UI is not the best like UML interaction diagrams in Eclipse who cares. Atleast I am getting some profiling data.

Maybe I'll try Netbeans as my main IDE once its released.

Fishy Serialization

In Java, "Object Serialization supports the encoding of objects and the objects reachable from them, into a stream of bytes". This representation of the object can then be used for purposes like persisting on disk or passing of objects over the network. To serialize an object the class needs to implement the Serializable interface which is a marker interface with no methods defined. Optionally the class may define "readObject" and "writeObject" methods which will be used in the serialization process as defined in Specification.

The signature of the read/write Object methods is what struck me recently(this implies that I've been reading serialization related code before and not realised this). The modifier is private which means that no other instance should be able to invoke that method! And yet its invoked somehow. Time for some hacking...


For a dummy class Dog the stack trace to call the writeObject from SerializeTest.main was:

java.lang.RuntimeException
at foo.bar.Dog.writeObject(SerializeTest.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at foo.bar.SerializeTest.main(SerializeTest.java:23)

The ObjectOutputStream.writeObject is invoked from the main method and after lots of calls ObjectStreamClass.invokeWriteObject is called which does some reflection The writeObject method is dynamically invoked there. The interesting part of the implementation is:

void invokeWriteObject(Object obj, ObjectOutputStream out) {
writeObjectMethod.invoke(obj, new Object[]{ out });
}


writeObjectMethod is a member variable in ObjectStreamClass of type java.lang.reflect.Method
and is set as:

writeObjectMethod = getPrivateMethod(cl, "writeObject",
new Class[] { ObjectOutputStream.class },
Void.TYPE);


The definition of ObjectStreamClass.getPrivateMethod is:

/**
* Returns non-static private method with given signature defined by given
* class, or null if none found. Access checks are disabled on the
* returned method (if any).
*/
private static Method getPrivateMethod(Class cl, String name, Class[] argTypes,
Class returnType) {
Method meth = cl.getDeclaredMethod(name, argTypes);
meth.setAccessible(true);
int mods = meth.getModifiers();
return ((meth.getReturnType() == returnType) && ((mods & Modifier.STATIC) == 0) &&
((mods & Modifier.PRIVATE) != 0)) ? meth : null;
}
}


And there is the call to the method which does all that magic - Method.setAccessible.

The javadoc for the method says:
"A value of true indicates that the reflected object should suppress Java language access checking when it is used"

Using Reflection and with proper access its possible to even call private methods. This was something cool that I've learnt in a long time. Certainly makes Java more dynamic in nature. Now I'll have to read more on the Java security API soon.

wget Google videos

Google has some amazing tech videos at their site by the user Google engEDU. And they even allow for downloads which is great for me.

But I prefer downloading files using wget to be able to resume downloads later. You can't just wget the video link url's as is because they have & characters which cause wget to try to download them as separate files. Instead try something like this for the "OSS Speaker Series: The State of the Linux Kernel" video..

wget -c --output-document=MortonLinuxKernel.mp4 "http://vp10.video.l.google.com/videodownload?version=0&secureurl=twAAAOFdafTKyCsBI7E0BCCT6060NjqUP0-3g9pfM0xl5X1YO8a1zhU5ArUNYf8PLb44VqTIrTR2hntorTVWAEL6bqWkChEIIVPqNeHV5F4PRqoHXwlvZRC0_giNoVtliPIDsfE7zAzQSPok2b8ShvvgxJVI3T3WoPwGtr6Vvmwfjj18i9wTbaZD_JGXpTjD2kJrvVIenN5CSTRBbXYMKR49YyVYLdRNu9BuY924qZaDV4zEd9YFUsoCP42JkguszqVlOg&sigh=hEHjkZDlhZrMiZnAQrs8O55mFdo&begin=0&len=4897661&docid=1742374580386548257&rdc=1"

Playing with Javascript

I've pasted an early version of a recursive function to walk the DOM from a particular node in Javascript. If you run the code though.. your browser will hang as it goes into a recursive loop. So whats wrong with the code?

<html>
<head>
<script type="text/javascript" language="javascript">
function walk(node) {
if (node) {
//do something with node
for (i=0; i<node.childNodes.length; i++) {
walk(node.childNodes[i]);
}
}
}
</script>
</head>
<body onLoad="walk(document.body)">
<a href="http://www.parivartana.org">parivartana.org</a>
</body>
</html>

The variable i in the for loop should be declared as 'var i' to fix the behaviour. Thats when I realised that I should not really code in Javascript assuming that it is a subset of Java. (But being as lazy as I am never really studied the language).

Then I came across some fantastic videos on Javscript by Douglas Crockford at the YUI theatre which is a part of the Yahoo Javascript library YUI. Some of the features of the language are explained really well. Some key features are Objects as containers, Prototypal Inheritance and Lamda. He also explains some browser and Javascript quirks.

Now I am wondering if Javascript should be a part of my resume!

One of his recommendations is to favour minification ie removing of whitespace to reduce download size versus obfuscation. Google actually always heavily obfuscate their Javascript. Also as part of the Google Web Toolkit, the deployable code is also obfuscated. So thats a debateable topic. Time to read some GWT generated code then.

Re: Select you crazy Query

Are you saying the first query is wrong cause it did not get the results you expected? Cause I don't see anything technically wrong with it. A left outer join does not mean that at the end of the query (whatever it may be), you will get all foos. What you get in the end depends on your WHERE clause.

Break the first query down and you will see how it was arrived at. The cross product of the tables based on the ON clause of the LEFT OUTER JOIN "did" contain foo(1) but the WHERE clause which was applied over the cross product eliminated that record. Hence the result.

The behavior is standard and not MySQL specific. Both those queries on a different engine should return the same behavior. Also, I don't see why the ON clause should remember the composite primary key.

More on left outer join here.

Select you crazy Query

Lets try a small SQL quiz this time. We have two tables foos and bars with the definition below..

CREATE TABLE foos ( foo_id INT NOT NULL, PRIMARY KEY (foo_id) ) ENGINE=InnoDB;

CREATE TABLE bars ( foo_id INT NOT NULL, bar_id CHAR(1) NOT NULL, PRIMARY KEY (foo_id, bar_id), FOREIGN KEY (foo_id) REFERENCES foos(foo_id)) type=InnoDB;

And lets do some sample data inserts..

INSERT INTO foos VALUES (1), (2), (3), (4);

INSERT INTO bars VALUES (1, 'a'), (2, 'b'), (3, 'b'), (3, 'c');


Now write query to give me all foos and those bars who have a bar_id 'b' for the same foo_id. Seems like an easy OUTER JOIN. This is what I came up with initially..

SELECT * FROM foos LEFT OUTER JOIN bars ON foos.foo_id = bars.foo_id WHERE bar_id = 'b' OR bar_id IS NULL;

And the result set was..

foo_id	foo_id	bar_id
2	2	b
3	3	b
4	NULL	NULL
3 rows in set (0.00 sec)

But the result set obtained is wrong because we did not get all the foos. Get back to the query then to obtain the result set below..

foo_id	foo_id	bar_id
1	NULL	NULL
2	2	b
3	3	b
4	NULL	NULL
4 rows in set (0.00 sec)

For the correct query just make a minor change to the above SELECT

SELECT * FROM foos LEFT OUTER JOIN bars ON foos.foo_id = bars.foo_id AND (bar_id = 'b' OR bar_id IS NULL);

I did not have enough time to search for the actual reason for this behaviour though and am not even sure if this is standard or MySql specific.

In table bars the primary key is a composite key between foo_id and bar_id. When foo_id is compared in the ON clause, it appears as if the rest of the primary key is forgotten. foos (1) joins with something like bars (1, NULL) and so the WHERE clause fails. When the ON contains all the clauses for the whole composite key, the LEFT OUTER JOIN behaves as expected.

That gives a whole new perspective to outer joins when dealing with composite primary keys then.

[TrackBack]StartUps in India

Santosh on StartUps in India

Java is now under GPL

That was really some amazing news. Yesterday was like a dream thinking of how bold Sun had been. Later I was thinking of what this really means for the language.

The obvious things are better community interaction with things like bug fixes and ports(palm anyone??).

Java SE will soon be bundled in almost all Linux operating systems. I think Gnome was using some Mono based applications. Now they could actually have the Java VM distributed. A whole new range of Java based desktop apps will be released. Its important to note that version 7 has been released now and version 6 will also be released later. That means that all free implementations of Java will use version 5+ Api's and so the fancy features like Generics, Enums and foreach etc. Java 6 also has a whole range of desktop specific enhancements so the "feel" of Java apps should also improve.

I am not sure if it is possible to deploy the Java ME VM without paying Sun any license fees. I want Sun to make money of Java, but if they can deploy Java to most phones/devices that would be great. Could this be the next platform after all.. almost making Java appear as an Operating System?

Of late Java lost a bit of steam. The language is mature and there are a large number of libraries to help in most tasks but age is also showing. Dynamic languages like php and ruby(with rails) provide features for faster webapp development. Even C# has some really cool features. This announcement will provide some extra steroids. In the future it will be interesting to see how "dynamic" Java can get.

To conclude.. Sun just made a lot of hackers really happy. This will definitely have some positive effects for the company and the language.

High on Emacs!

For the past three weeks I have been playing around with Emacs. Right from learning how to do simple editing to finally getting the Java Development Environment for Emacs (JDEE) setup on my machine, its been a whirlwind tour of this awesome editor!

I used to be a big "vi" fan, but the sheer amount of work that Emacs allows you to accomplish is simply superb! I'm looking at a phase of experimentation, to see if Emacs will be productive for me on a daily basis. For now, the answer seems yes!

Check out:-

GNU Emacs

Emacs Tutorial

Learning GNU Emacs

JDEE

Happy editing!

His name is Linux

Re: Free the Platform!!

Looks like someone else had the same itch



Yet another link to the presentation above here. Get more info on the Linux phone here and here. You can get some more information on the State of Linux phones

THERE HAVE BEEN a lot of phones claiming to be 'Linux phones' and those that do run a Linux kernel, but they all miss the point of Linux: to be open. FIC is about to change that in a big way with a truly open phone, the OpenMoko.

Some of these phones are already in the market from Motorola and others. Companies like WindRiver and MontaVista also make Linux based stacks but I am not sure how open they are and whether it is possible to change/upgrade everything.

Within a few years more and more companies will release mobile frontends to their webapps. Just as Gmail recently did. As of now Symbian, Windows CE and Palm and the main OS'es for smart phones. Who then will win the battle for the dominant platform? Java has an enviable position right now as it has already been deployed in many phones already. If Sun opensources Java under GPL then there will be even more widescale adoption in the open source stacks. Nokia Series 60 also allow programming in C++ and Python now so thats another race.

Free the Platform!!

Can anyone tell me why I cant upgrade my mobile phone software?? Not some Java application on the phone, but the actual operating system. How much of a difference is there between a Nokia series 40 and Nokia series 60 phone in terms of hardware? Should I not be able to swipe the existing JVM which was based on an older Java implementation to a newer one? Will I ever be able to customise the hardware in the phone? Maybe we'll have to wait a bit longer for programmable hardware.. probably through FPGA's.

In a few years the mobile phone will be more important the the average computer. More people will use handheld devices for their daily tasks like browsing the net and listening to music. Developers will follow.. creating the next generation of killer applications.. followed by hackers to free the platform.. and by then google will own all your data

Ubuntu 6.06 (Dapper Drake) on Acer AS3004WLCi

I had installed Dapper Drake on my Acer AS3004WLCi Laptop a few months back. I've been happy with it but there were several things that didn't work. I've been pretty lazy but I think I've finally resolved most, if not all, of them. Here's a run down.

Firstly, what didn't work:

• After booting up and picking Ubuntu from the grub menu list, a few kernel messages would show up and then there would be a blank screen until X Server started up with the login box. Similar issue when shutting down... didn't see any messages.


• Same thing for Virtual Terminals... Ctl-Alt-[F1-F6] would bring up a blank screen.


• Resolution for the 15.4 inch screen would only bring up 640x480, 800x600 and 1024x768 options. No wide-screen 1280x800 resolution.


• Touchpad scrolling would not work.


• Suspend/Hibernate would work the first time after booting up, but after resuming there would be a message saying there was a problem and on subsequent attemps would fail. It would look like it's doing something, screen would go blank, then all of a sudden some garbled text would be displayed at the top of a black screen and the screen saver would start up.


• Wireless would not work. Would not detect the card.


• Issues with sound... If multiple applications which use sound such as a music player, a flash site and skype were open, things would be very erratic. Skype would give errors like 'Problem with sound device'. Flash based sites like YouTube or Google Video would not have sound etc...

What I did to resolve these issues:

• Turns out that the blank 'bootup/shutdown screen' and blank virtual terminals are related. When you pick Ubuntu from the grub menu list, it loads up the linux kernel which in turn loads a basic video driver to show the kernel messages. The video driver is considered to be basic and compatible with most screens. If the vga parameter is not passed to the kernel it defaults to vga=normal. It seems it wasn't with my Acer screen. So this needed to be changed. Edit /boot/grub/menu.lst. Add vga=792 to the kernel /boot/vmlinuz-<version> root=<hda> ro quiet splash line.
  
  HOWTO: Change bootup resolution was really helpful. It goes into further details about the bootup process etc... Also read it to understand the 792 value.
  
  The kernel uses the same driver for both the 'bootup/shutdown screens' as well as the virtual terminals. The vga parameter fixed both these issues.


• The touchpad scrolling problem and the screen resolution problem had a common resolution.
  
  lspci | grep VGA
  shows
  VGA compatible controller: Silicon Integrated Systems [SiS] 661/741/760/761 PCI/AGP VGA Display Adapter
  on my Acer laptop.
  
  So we need a sis driver. Looking at /etc/X11/xorg.conf showed that it was loading the vesa driver. To fix this run
  sudo dpkg-reconfigure xserver-xorg.
  
  Go through all the screens and answer the questions. For most, the default will do. When you get to the video screen, pick the sis driver and the desired resolution (1280x800). Make it the default. After this is done restart X-server (Ctl-Alt-Backspace). It should start with the crisp widescreen resolution.
  
  The Strange screen resolution thread got me the lead on trying dpkg-reconfigure.
  
  Check this page for some bedtime reading on sis drivers.
  
  The touchpad scroll started working after going through the reconfigure. I didn't do anything specific. I think I just picked default options for the touchpad/mouse screen.


• The Suspend/Hibernate issue was related to Wifi not working. Funny how these things are related no? The garbled text that was displayed after a failed suspend became clear after the virtual terminal issue was resolved. It was outputting bcm43xx: Error: Microcode "bcm43xx_microcode5.fw". dmesg showed the same error.
  
  lspci | grep Wireless
  shows
  Network controller: Broadcom Corporation BCM4318 [AirForce One 54g] 802.11g Wireless LAN Controller (rev 02)
  on this Acer laptop.
  
  So this is the wi-fi driver. After some searching, came upon How to: Broadcom Wireless cards (These How-Tos are amazingly helpful!). I followed the instructions and it seemed to work. iwconfig brings up the wireless network interface information + iwlst <eth> scanning shows the wireless connections available.
  
  After suspending and resuming, looking at the dmesg output, the errors weren't there anymore.
  
  Check this page for some insight into the suspend/hibernate process.
  


• The sound issues took care of themselves really :) Upgrading the kernel to 2.6.15-27-386 seemed to resolve all of them. Also upgrading to Skype 1.3.0.53 helped.

Most problems have solutions out there. All it needs is a bit of searching/reading/trying.

Lets see how many of these issues are resolved in Egdy Eft which comes out tomorrow!

Dependency biculturalism

Nicely put: What do dependencies have to do with Free Software?

This relates closely to Biculturalism which Joel talks about.