Tuesday, December 30, 2008

Pure Idea Joy

One of the reasons why I love IntelliJ Idea. Look at the text in red below.

package com.msh.ui;

import java.util.ArrayList;
import java.util.List;

/**
* User: sachinh
* Date: 28-Dec-2008
* Time: 17:22:38
*/
public class ParsedCommand {
private List<Property> props = new ArrayList<Property>(5);
private List<PluginInfo> mojos =
new
ArrayList<PluginInfo>(5);

public ParsedCommand(String cmd) {
parseCommand(cmd);
}

public List<Property> getUnusedProperties() {
for (PluginInfo p : <ctrl+shift+space> fills in mojos) {

}


Just brilliant - the fact that it picks up the right collection field
based on the Iterator's type. I think its called Smart Code Completion.

Thursday, August 02, 2007

Rising to the top

Recently, I posted a one line, grammatically incorrect (actually it just had a typo - a missing 'of') blog in which all I did was praise Seth Godin's article and point to it. I didn't think anything I had to say about the topic could really add any additional value so I refrained from putting in any of my comments. I didn't think much about it then but afterwards I found out, interestingly, this article had reached the top of the Popular Entries for the last 24 hours at JavaBlogs.com. This is not really such a big deal but I have an RSS feed to the popular entries and it is always interesting to see what rises to the top in that list. Because Mozilla's Live Bookmarks shows only the title, I have to judge whether I want to read the article based on the title.

Despite it being a collection of Java blogs, it is infrequent that anything seriously technical about Java manages to hold popular attention. It is always the more interestingly worded title that starts to move up rapidly (perhaps the only exceptions are the latest buzzwords and the word 'performance'). More technical titles generally languish somewhere in the middle of the list which at the top is crowded by articles that are inflammatory in their title or simply very vague about their content. The advantage of the latter (which mine must've inadvertently become) is that you have to click on the link to find out what's in the blog. This makes the blog more 'popular'. This, of course, makes it go up higher in the list allowing reaching more people and setting off a chain reaction.

So, here's (deliberately this time) a rather vague title for this blog to test this theory :-)

Tuesday, July 31, 2007

Managing Expectations

Seth Godin has this truly wonderful blog which is always very interesting to read and this piece advice is true gold.

Wednesday, July 18, 2007

Another reason to have X-GOOGLE-TOKEN?

In my previous post here about XMPP I had linked to a blog (here) that describes the X-GOOGLE-TOKEN mechanism of authentication. dJOEk asks a question "Why does Google use a proprietary authentication mechanism" and then goes on to make a point about how X-GOOGLE-TOKEN could be one of the first steps twoards a Single Sign On solution from Google. This point of view has received a lot of coverage with a several people commenting about the merits of this idea or its feasibility.

I have recently noticed another possible reason for X-GOOGLE-TOKEN to be made available and this is what I am putting forth here -
I had written in my blog that when I tried to sniff out the conversation between Google Talk and the server using Ethereal I found that it was using TLS and not X-GOOGLE-TOKEN and therefore all conversation was encrypted. Several people have since asked for ways around this but the whole point of TLS is to prevent such sniffing and decryption is practically (at least to me) impossible.

Since then my current company has installed an auditing software (I know :-() for compliance reasons and interestingly Google Talk is back to using X-GOOGLE-TOKEN. When using X-GOOGLE-TOKEN only the authentication part goes over TLS while the rest of the conversation does not which means that conversations are in plain text and can be intercepted, audited and archived. So, another possible reason for supporting a different authentication mechanism could be to be able to support auditing and monitoring software?

Friday, June 29, 2007

Progress job posting on Joel India Jobs Board

Joel from JoelOnSoftware has this jobs board specific to India jobs which we are trying for the first time. The idea is that we will either get a bunch of resumes from the kind of people we like to hire or hear from or we will not get anything at all. Our usual experience from recruitment agencies and job sites like Monster is that we get a lot of resumes (with less than 10 percent being the ones we seriously consider and interview) and sifting through them just kills us.

Here's the link to the job post http://jobs.joelonsoftware.com/?2202

I would love to hear from people who have tried the Joel India Jobs Board and hear their experiences. If this works for us we'll do more job posts here.

This is the text of the job post -

Software Engineer

at Progress Software

Hyderabad, India

We are a small and focussed engineering team in Hyderabad developing the next wave of SOA development tools for our product, the Sonic Enterprise Service Bus. We have been involved in writing Eclipse based high-end editor plug-ins for open standards based technologies such as XML, XPath, WSDL, etc using a variety of Eclipse frameworks such as EMF, GEF, etc. We have developed code generators and graphical tools for generic mapping capabilities, developing BPEL, etc, amongst many others.

We are looking for smart and competent people to join and enhance the experience of SOA development for our users. The next few releases are going to be very exciting as we add features to improve developer productivity, including refactoring distributed applications, visual development, and on-line assistants.

We are looking for excellent programmers in Java with up to 3 years in experience who also possess these qualities -
Creativity - we like people who can think of new features for our products and make a difference to our users
Capability - we want people who are very good at programming and can demonstrate it to us
Self-driven - we are self-driven people who are expected to work with minimal supervision and get things done

Any in-depth Swing or Eclipse development experience is a plus but not a requirement.

The team is completely based out of Hyderabad and has a QA to developer ratio of over 1-to-1. We participate in local college internship programs and in various forums.

Comments about the Joel Test: We fix all test blockers before we write new code. The "quiet working conditions" implies offices as per the Joel Test - we have workspaces and we like to mingle and have some fun.


Our interview process is heavily biased towards programming skills so please apply only if you are serious and think of yourself to be an excellent programmer.

Joel Test Score: 11/12

The Joel Test is a twelve-question measure of the quality of a software team.

Yes! Do you use source control?
Yes! Can you make a build in one step?
Yes! Do you make daily builds?
Yes! Do you have a bug database?
Yes! Do you fix bugs before writing new code?
Yes! Do you have an up-to-date schedule?
Yes! Do you have a spec?
No. Do programmers have quiet working conditions?
Yes! Do you use the best tools money can buy?
Yes! Do you have testers?
Yes! Do new candidates write code during their interview?
Yes! Do you do hallway usability testing?

Interested?

If working with a small, focussed, informal team that is completely engaged in every aspect of product development starting from product direction and requirements gathering to engineering excites you then send us your resume at joeljobs@progress.com

Friday, March 09, 2007

Who cares about the chronological ordering of blogs?

There is a new way of designing blogs at Blogger and in general it is very easy - point and click, move around, group - the usual goodies associated with good Web-UI these days. The new blogger re-design also has a different mechanism of listing previous posts. They now group it by chronological order. I have spent some time trying to understand why this is useful to people who come to my site? And I have been completely defeated in understanding its purpose. Before I go into a litany of my issues with this new way of organizing posts I want to show what I mean -

This is the old way it used to be -

And this is the new format (which you can see in the right pane of this page if you are at the original blog site and not seeing this through a reader) -

I don't see why this useful for anyone other the blog author - and even for the author it is only informative (ah February 2007 was a good month for me - Oh Gosh! I didn't post a single thing between October 2005 and August 2006).

Here's what is lost by going to this format -
a. Earlier visitors (especially first-time visitors which are the majority of the visitors according to Statcounter's indication of the abysmal popularity of this blog :-)) cannot see a quick list of other posts which might catch their attention. They now have to do a deliberate task of expanding those date nodes to see what I might have written previously. I might just stop putting in titles altogether.
b. And really what is the reason to group it by month or date - do any of my readers really relate to the dates that I published my articles? How is August 2006 any more important than Jan 2007?

I am sure that there is way somewhere to hack out of it and maybe I'll have to sit down and hunt for it. I searched for it in the options and could not find a way to switch this off.

Monday, February 26, 2007

Accepting the Q factor

http://www.w3.org/2000/09/xmldsig# is the namespace for the schema for XML Signatures - one of the many, many schemas you end up accessing if you do XML Schema based completion for WS-SecurityPolicy (2005) (part of our WSDL policy editor in the Eclipse plugins for Sonic ESB Workbench). Why is this one special? For the following reason -

If you access http://www.w3.org/2000/09/xmldsig# from Mozilla Firefox you will get back the schema at http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/xmldsig-core-schema.xsd (through an HTTP re-direct response code 303) but if you use Java's java.net.URL.openConnection() (basically through HttpURLConnection) you get an HTML page and not the Schema (XML) which our Schema loader does not particularly appreciate.

It took a while for me to understand why the same URL is behaving differently. Using Eclipse 's TCP/IP Monitor I captured the headers sent by my code and used LiveHTTPHeaders for Firefox.

This is what Firefox sends -

GET /2000/09/xmldsig HTTP/1.1
Host: www.w3.org
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

and this is what it receives -

HTTP/1.x 303 See Other
Date: Mon, 26 Feb 2007 11:50:20 GMT
Server: Apache/1.3.37 (Unix) PHP/4.4.5
WWW-Authenticate: Basic realm="W3CACL"
Location: http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/xmldsig-core-schema.xsd
Keep-Alive: timeout=2, max=99
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1



But when HttpURLConnection sends the request this is what it sends -

GET /2000/09/xmldsig HTTP/1.1
User-Agent: Java/1.4.2_12
Host: www.w3.org
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive


and this is what it receives -

HTTP/1.1 303 See Other
Date: Mon, 26 Feb 2007 11:56:47 GMT
Server: Apache/1.3.37 (Unix) PHP/4.4.5
WWW-Authenticate: Basic realm="W3CACL"
Location: http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/Overview.html
Keep-Alive: timeout=2, max=99
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1


Notice the difference in the Location header
Firefox : Location: http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/xmldsig-core-schema.xsd
Java URLConnection : Location: http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/Overview.html

The problem turns out to be in the Accept header set by Java URLConnection by default (or I guess the Sun HttpURLConnection implementation).

Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2


Note no text/xml as in Firefox. Although there is a */* its 'q' value is lower than text/html and the nice server at www.w3.org uses this to change its output to suit what is best accepted by the user-agent. I guess since they are the standards organization they should do this :-). Fixing the accept header fixes this behaviour. Is there something I am missing in my understanding of how URLConnection works?

Saturday, February 24, 2007

Experiments with development on the Nokia 6265

I recently got myself the Nokia 6265 CDMA phone. It is a nice phone and I intend to write a reveiw of it in my other blog one of these days. The phone supports the Nokia's Series 40 3rd Edition development platform and I am trying out developing Java Micro Edition applications for it. I hit some interesting problems and found some solutions - I am trying to chronicle them here.

Firstly, I have never done development for a mobile phone so I had to start from scratch. I wanted to know what I could do with my phone and Nokia had quite a lot of information. They have these videos of their Eclipse integration called Carbide.j which seemed interesting although a little painful (being a graphical modeling like UI and all).

I found the Nokia site a tad vague about what exactly is needed for what but eventually I figured I needed the Nokia Series 40 3rd Edition SDK. Now, Nokia has 3rd Edition Feature Pack 1 and 3rd Edition Feature Pack 2. It wasn't very clear to me which one would work and I chose the base version and thankfully that is the right one. Only one phone so far supports Feature pack 1 as far as I know.

After installing the SDK, the documentation said that I need a JDK, Eclipse, NDS for Java ME, and then I should install SDK. Did I mention that the Nokia site is slightly confusing? :-) I went off in search for NDS and found out that it is now renamed to Carbide.j and is a 135+ MB download. While it was downloading, I thought I would try and locate J2ME support in IntelliJ Idea.

IntelliJ Idea J2ME integration

Now, this is the nice part.

I went through the IntelliJ Idea documentation which described how to set up a J2ME module and the only part that looked a little hard was the Mobile JDK configuration. I crossed my fingers and pointed at the Nokia installation and IntelliJ found all it needed - bootclasspath, javadocs, emulator, etc. It is really sweet!!

Everything works out of the box - compile works great, and when you run it runs against the Nokia emulator. Even debugging is seamlessly supported!! Fantastic - and no need for Eclipse or the Eclipse plugins! Needless to say I cancelled my Carbide.j download. You do not need it for development on the Nokia phones. I hope the Nokia guys put this on their documentation.

One strange problem so far (in my HelloWorld application) has been that the jad and jar generated by IntelliJ is not working on my phone or on the emulator. It fails with an error message saying "Application invalid. Delete?". There is a temporary file that Idea is generating which seems to work. I think I am missing something here. I would appreciate any help.


Annoying UI Change in Firefox 2.0

Firefox 2.0 is out and I upgraded from 1.5.x to 2.0. While closing a tab I realized something was amiss. In 1.5.0 there are two ways to close a tab - each tab is associated with a close button and the tab bar has a close button at its extreme right which is applicable to the most active tab. That is the button I use the most and that is gone from 2.0.



Why do I think this is an annoying change? The old button on the right of the task bar never moved - which meant irrespective of which tab I was on I could aim my mouse over it (a centimeter square area that my hand and wrist manage to reach magically) and close without having to think too much about it. Now, I have to look for the close button and my target has just expanded horizontally to unpredictably include my entire horizontal resolution. Another reason was I could close many consecutive windows one after the other just pressing the button multiple times because the selection automatically moves to the next tab. No longer possible. Arghhh! Hopefully there is a preference somewhere which can get it back - but so far I haven't found it. In its place there is a drop down of all windows to quickly navigate to it.

Well...sigh...I still love Firefox and I'll just use Ctrl+F4 more frequently...

Sunday, January 14, 2007

What I don't like about the iPhone

Apple has hit that elusive textbook-class marketing success. iPhone has the same buzz that Gmail had when Google decided that their Beta product is available only by invitation. iPhone is 6 months away from release but look at the talk around it. Everyone has heard of it now because everyone wants to talk to someone else about it . And everyone is blogging about the iPhone - and here's my two cents on it.

You've heard the comments - it is a gorgeously styled phone but has not much new to offer - it is just slickly packaged and marketed. It is quite possible that the actual product might be fantastic. A lot of the magic is in the User Interface and there is only so much you can say about it - like the Apple iPod. The fact that the click wheel makes a clicking sound (by the small speaker only for it) is best when used and not when you read it. So, final judgements on how great it is can probably only be passed once the product comes out. It does look very nice.


So, what is this blog about - just a couple of points about the UI from the demos I saw on the apple site.

Take a look at this screen - I know the iPhone does not have a keyboard and provides a software keyboard which you can use like a regular one. That is a fine idea but look at the keyboard.

What I don't like about it is that although it is a QWERTY keyboard it does not have a full keyboard - the digits and symbols have to be accessed much like an old phone which had menu options for 123, Symbols, etc. Unfortunate.

Blackberry keyboard and screen togetherThe next thing that strikes me is that there is not enough space to see or read what is on the screen after the keyboard pops up - most of the screen is taken up by the keyboard and the effective space is down to a couple of lines. My age-old phone has more space than that. My wife's Blackberry (left - I couldn't capture a better image of email and keyboard together - it isn't meant to be Blackberry marketing! :), despite its not so good-looks, has a practical amount of screen real-estate. Definitely a win for the Blackberry here. Usually, phones in this category support hand-writing recognition using a stylus which allows them to provide larger screen estate without the keyboard in the way or they do it the Blackberry way. With no stylus the iPhone loses space.

This is the SMS screen (right). Looks cute but if the demo follows the product faithfully the ordering of messages seems wrong. I believe it should be in reverse chronological order so that the most relevant messages (the recent ones) are on top versus having to scroll down. Also, this is perhaps the most wasteful of screen layout designs - nice but wasteful. Hardly a good idea considering how little of it you have. But then perhaps these are just preferences that you can set - like Google Talk.

What I like is the interface you get when you are in a call. The large buttons that come up for Speaker, Mute, Hold, etc are excellent for usability. I have the hardest time figuring out which button is which on my phone and even on the Blackberry you have to know what button does what - this is so much more elegant and usable.

All said, the Apple iPhone is definitely going to be a serious contender in the phone market and like in the case of music players might cause the other competition to provide a host of new features. Most people have predicted that the iPhone will not have the same impact that iPod did - which is probably very easy to predict because the phone market is quite mature and a tie in to a service provide (Cingular) restricts the user-base but I still expect the iPhone to make quite an impact. Whether the Blackberry users switch - can't say - those things are quite addictive.

The overall iPhone UI theme is large, informal text which looks aesthetically pleasing and is very readable but wastes space. In contrast, business phones like the Blackberry are all business-like keeping everything to efficient levels while compromising on looks, jazz and style.

Btw, I wonder if I am the only one who finds QWERTY keyboards a strange feature on mobile phones when people type with their thumbs. So quickly now - Where is the D key, the P key, or the N key, the X key - can you answer without looking at the keyboard? For people who can type with all fingers it is quite useless because the fingers keep the memory of the letters they type and not the thumbs (in fact the thumb only knows the space key) and definitely not the eyes so they have to hunt for the letters. For people who can't type and look at the keys when they type, I wonder if they do they find QWERTY layout any easier than a ABCDEF layout? Seems to me that for either group the ABCDEF layout makes more sense than having to hunt the entire screen.

Tuesday, October 24, 2006

Part 3: Jabbering with Google Talk over XMPP

This is part 3 of a series of blogs on getting IM working with a Google Talk client using XMPP. Parts 1 and 2 are here -
Jabbering with Google Talk over XMPP
Part2: Jabbering with Google Talk over XMPP

This one is going to be short one because I managed the last part of the Google Talk basic IM interaction which was presence and messaging without too much effort.

Presence
Presence is how you announce that you are available/unavailable/busy/whatever. It is through presence that people tell you that they have stepped away for a quick cuppa mocha - the part where you can have witty remarks show up against your name - basically your network availability.

The basic presence part is easy - requiring sending only a stanza. If you are already on the "roster" (XMPP's name for a buddy list) then your presence will be indicated by your id going "online" in Google Talk. You would have to be invited and on the users' roster for that.

Doing the presence part itself is optional - you can still send messages even if you have not announed your presence. The only difference is that messages directed to you are not sent by Google Talk because it is not aware of your avaiability and your friends won't know if you are online.

Presence has a bunch of interesting details which I won't go into mostly because I skipped most of them :-) but here's a quick list of what it can do -

  • announce availability/unavailability plus a bunch of sub-states
  • provide comments along with status
  • allow for subscription (notification of presence of others on the server)
  • change subscription
  • etc
Instant Messaging
The whole IM bit is where I was leading through and here it is. I did cover in Part 2 the messaging part which was basically

<message from=\"XXXXXXXX@gmail.com/D922F673\" to=\"YYYY@gmail.com\" type='chat' xml:lang=\"en\"><body>TESTING!!!</body></message>

and this is pretty much it actually. You can have the following in the message though - subject(s), body and a thread.

So once you have set up a Thread reading from your response stream you get message stanzas from all your friends and you can send them your messages on the request stream - thats it. Google Talk behaves just fine identifying the program as a non-Google Talk client.

Here's a silly picture of the two clients talking to each other.

Saturday, October 07, 2006

Answers to some seemingly common Java questions

Whenever I go through the Keyword Analysis page of Statcounter, which manages my blog web usage statistics for me, I see a bunch of Java questions which I can answer. However the page that people land on to never has the relevant answer to the query. With my recent Repetitive Stress Injury I am pretty much staying away from doing any more work than is required to help my hands heal faster. This is keeping me from working on my Google Talk programs. So, having nothing better to do I am going to try and answer some of the questions or searches that I saw coming to my blog.

Search Keywords: Java printStackTrace does not show line number
Happens if the classes that are part of the stack trace have not been compiled with the debugging option (-g) on. Here's the link for details on javac's options. The solution is to re-compile the classes with this option on and then recreate the exception. If it is not code you can build then there is not much you can do here unfortunately - you will have to analyze the code to figure out where it went wrong. Not having debug information while compiling usually also means that you won't get any local variable information while debugging in a JPDA debugger (to be complete precise that would mean that the -g:vars option has not been specified).

Search Keywords: how can i tell what caused a concurrentmodificationexception?
A ConcurrentModificationException happens if a java.util.Collection is modified while an Iterator is iterating over it. There are many ways you can end up doing this - I am going to try and list the situations that I believe are most common.

Disclaimer: Code snippets do not use generics which won't make any difference anyway.
1.] Listeners
Consider this listener interface -


XYZListener.java
1 public interface XYZListener {
2 void eventOccurred(Event evt);
3 }



and this implementation -

XYZListenerImpl.java
4 public class XYZListenerImpl.java {
5 public void eventOccurred(Event evt) {
6 evt.getEventSource().removeXYZListener*this);
7 // do something useful
8 }
9 }

where Event.getEventSource() returns the object against whict XYZListenerImpl's instance was registed as a listener.

Now, you will get a ConcurrentModificationException if the EventSource is implemented this way -

EventSource.java
10 public class EventSource {
11 private List listeners = Collections.synchronizedList(new LinkedList());
12 public void addXYZListener(XYZListener l) {
13 if (!listeners.contains(l))
14 listeners.add(l)
15 }
16
17 public void removeXYZListener(XYZListener l) {
18 listeners.remove(l);
19 }
20
21 protected void fireEvent(Event evt) {
22 for (Iterator iter = listeners.iterator(); iter.hasNext();) {
23 XYZListener listener = (XYZListener)iter.next();
24 listener.eventOccurred(evt);
25 }
26 }
27 }

This is going to cause a ConcurrentModificationException at line 24 (assuming that there are more than one listeners registered and XYZListenerImpl is not the last one ;-)) because in line 6 the listeners List is modified while it is being iterated over in EventSource.fireEvent's Iterator.
The solution in this case is to use the right pattern for firing events -

protected void fireEvent(Event evt) {
List clonedList = new ArrayList(listeners);
for (Iterator iter = clonedList.iterator(); iter.hasNext();) { // [19 Dec] edited - thanks to Anon comment
XYZListener listener = (XYZListener)iter.next();
try {
listener.eventOccurred(evt);
} catch(Exception ex) {
// this prevents one bad listener from preventing the event from going to others
// log the exception
ex.printStackTrace();
}
}
}

2.] Incorrect coding
This will cause a ConcurrentModificationException -

public void someMethod(List l) {
for (Iterator iter = l.iterator(); iter.hasNext();) {
Object o = iter.next();
if (someCondition()) {
l.remove(o);
}
}



Fix this by using Iterator.remove() instead of doing a list.remove()

3.] Concurrency
The trickiest one is when the Collection gets modified by another thread while it is being iterated upon. One solution is to clone the collection (e.g. new ArrayList(list)) and then iterate upon it. To find out where the List got modified -
  1. Create a wrapper List similar to Collections.SynchronizedList which delegates all methods to the enclosed List.
  2. In the add/remove and any other method that modifies the List dump the current Thread's stacks using Thread.dumpStack. Refer to this. I suggest printing the current timestamp and the Thread's id for easy collating and the List's hashCode() to identify operations against each List instance.
  3. When you get the ConcurrentModificationException print the List's hashCode and look for the last print from the List modification logs for a list of this hashCode and you should know which two threads are the "culprits".

Search Keywords: how do you add a print statement ever 5 minutes in java
This is quite easy. Create a java.util.Timer class and add a TimerTask and set it to fire every 5 minutes and write the print statements in the TimerTask.

Search Keywords: using ethereal to capture google talk
This is not really possible after TLS is set up as I discussed in the comments here. Google Talk mandates TLS and once the stream gets encrypted the whole point of that is to not be able to sniff out the contents using something pretty much like Ethereal.

There are a couple of other interesting queries that I have not taken up
what are some things that made java so popular?
java 5 why
(I would say - Generics)
why isn't java used for games
(I really don't know if it is or it is not used for games and what kind of games?)


Saturday, August 26, 2006

Part 2: Jabbering with Google Talk over XMPP

This is Part 2 of the posts that describe my attempts to do something interesting (eventually) with Google Talk using XMPP. Part 1 is here.

So finally I had my test Google Talk account successfully send a test message to my regular Google Talk account. The XML for that is

<message from=\"XXXXXXXX@gmail.com/D922F673\" to=\"YYYY@gmail.com\" type='chat' xml:lang=\"en\"><body>TESTING!!!</body></message>

Note that the from value is the JID returned by the Gmail server.


I am still quite far from where I want this to go. However, I am glad I am making progress. The last time (in Part 1) my quickly-put-together test code was reaching a point where it could no longer be used because I needed to save conversation state like the jid which it didn't allow.

Now, with a design and a framework in place I hope I can move on to newer things with Google Talk. Here's a brief class design diagram. I am not very formal with UML diagrams - I pick and choose what I like from UML so if it is not classic UML I apologize to those who may get irked by it. Let me know what you think of the design. I haven't seen what the Jive XMPP library design looks like yet.


XML Parsing Challenges
An interesting implementation challenge was XML Pull Parsing. I tried (briefly) the Stax parser in J2SE but I found it to be a little inconvenient with its event id based mechanism. I might be quite wrong though because I am sure I didn't spend as much time exploring its fit into my solution as I should have. I found myself thinking in terms of iterating over available pieces of XML as required.

The XMPP response stream is like an XML document where each response is another child of the document element. And you don't get the next child until you have sent a request. (I don't know how one receives messages yet so there might be some twist to this story later).

So, an XMLIterator which allowed me get the details of the current element and then waited until I asked it move to the next one was what I wanted and this open source project does pretty much that. Thank you very much Mark.

What I also wanted was an API that returned DOM nodes (preferably) as it saw them in the stream. Of course, the document element node will remain incomplete until the end is reached but thats just a technicality because all that is required to "complete" it is the end-tag which has no real information. I was looking for something like this -

XmlIterator iter = new XmlIterator(source);
iter.advance();
// we are now on the document element node
XmlIterator children = iter.children();
while (children.advance()) {
Node node = children.next();
}

I have a simple XML Node structure built over XmlIterator which works for me - maybe that's a project for later.



Next post I aim to be able to maintain a conversation with a Google Talk client and hope to have enhanced the framework to be able to do more things than just chat.

Monday, August 07, 2006

Jabbering with Google Talk over XMPP

I am writing an XMPP client that can talk to Google and maybe implement some cool things on top of it. So this weekend I began exploring the specs and intend to maintain an account of how it is going. XMPP is a widely discussed/implemented topic and has many many clients - so it is no research topic. Why did I choose to do this? Just for fun to try something new (for me) out. I know I can get the XMPP library from Jive Software but thats no fun. Sometimes re-inventing the wheel has its own pleasures :-).

The XMPP specs are an open standard on which Jabber and Google Talk are based. There are a number of extensions which are under consideration and most of the cool things have already been thought of such as RPC over XMPP.

So far as a prototype, I have managed to get connected to talk.google.com, perform starttls and authenticate myself followed by resource binding and initiating a session. Now, my prototype code is hitting its limitations and I will have to spend some time fixing it to be able to do some serious talking with XMPP. I have also managed to get my account successfuly blocked for authenticating incorrectly as well - but got out of that mess. :-)

I was hoping to sniff out the conversation that the Google Talk client is having with the server using Ethereal as per this blog that describes X-GOOGLE-TOKEN, a Google mechanism for Single Sign-on. As per that blog, the Google Chat client does not do starttls but does an XMPP authentication over an un-secure socket using a Google generated token (the actual authentication with the Google token server is over https) so all the communication can be sniffed. However, my client seems to be doing a starttls and I can't sniff any details out after the proceed response. Too bad - appears that Google Talk has changed since the blog was written.

Here's the sequence of communication with the Google Talk server - (formatted for readability with text sent from me in this colour and the response in this colour and comments in this colour).



<stream:stream
to='gmail.com'
xmlns='jabber:client'
xmlns:stream='http://etherx.jabber.org/streams'
version='1.0'>
<?xml version="1.0" encoding="UTF-8"?>
<stream:stream from="gmail.com" id="X0B367FC8A9597BA4" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">
<stream:features>
<starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>
<mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
<mechanism>X-GOOGLE-TOKEN</mechanism>
</mechanisms>
</stream:features>

<starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls" /> <--- Start TLS - basically the rest of the communication is over SSL
<proceed xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>

TLS Succeeded - we are good to go...

<stream:stream
to='gmail.com'
xmlns='jabber:client'
xmlns:stream='http://etherx.jabber.org/streams'
version='1.0'>
<?xml version="1.0" encoding="UTF-8"?>
<stream:stream from="gmail.com" id="X1A565C1E8E3FD7CA" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">
<stream:features>
<mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
<mechanism>PLAIN</mechanism>\
<mechanism>X-GOOGLE-TOKEN</mechanism>
</mechanisms>
</stream:features>

Now we get the PLAIN auth mechanism which is basically base64 encoded \u0000username\u0000password string which I have blacked out here.
<auth xmlns='urn:ietf:params:xml:ns:xmpp-sasl' mechanism='PLAIN'>XXXXXXXXXXXXXXXXXX</auth>
<success xmlns="urn:ietf:params:xml:ns:xmpp-sasl"/> <--- authenticated

<stream:stream
to='gmail.com'
xmlns='jabber:client'
xmlns:stream='http://etherx.jabber.org/streams'
version='1.0'>
<?xml version="1.0" encoding="UTF-8"?>
<stream:stream from="gmail.com" id="X77D6827CD0B365BA" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">
<stream:features>
<bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"/><session xmlns="urn:ietf:params:xml:ns:xmpp-session"/>
</stream:features>

<iq type='set' id='bind_1'>
<bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'/>
</iq>
<iq id="bind_1" type="result">
<bind xmlns="urn:ietf:params:xml:ns:xmpp-bind">
<jid>XXXXXXXX@gmail.com/D922F673</jid>
</bind>
</iq>


<iq to='gmail.com' type='set' id='sess_1'><session xmlns='urn:ietf:params:xml:ns:xmpp-session'/></iq>
<iq from="gmail.com" type="result" id="sess_1"/>

Authenticated, Resource bound and Session created. Now, I need to send a message!

</stream:stream>



Next article on this, I hope to have successfully sent a message.

Friday, July 28, 2006

Usability in MS Outlook Tasks

I really love the way the MS Outlook's Task List Due Date field works. You can input dates in a variety of fashions and I am not talking about the MMDDYYYY, DDMMYY formats. Here are a few examples -

So if today is 28 July 2006 (Friday)
You can type

  1. W - (next week) - 4 Aug 2006 (Friday)
  2. 2W - (n weeks from now) - 11 Aug 2006 (Friday)
  3. Y, nY - (n years from now) - 28 July 2007 (Saturday)
  4. 3D - (n Days from today) -31 July 2007 (Monday)
  5. Aug - (next August, and other months work) - 28 Aug 2006 (Monday)
  6. Monday - (next Monday) - 31 July 2006 (Monday) (works for all the days of the week)
  7. Today - 28 July 2006 (Friday)
  8. 29 - 29 July 2006 (Saturday)
  9. 2007 - 28 July 2007 (Saturday)
  10. Aug 3 - 3 Aug 2006 (Thursday)
  11. 2 Wed - (n Wednesdays from the next one :-) ) - 16 Aug 2006 (Wednesday)
Some behave unexpectedly e.g. 3 Aug gives me a 3 Aug 2007 while 16 Aug gives me 16 Aug 2006. Not sure why.

M also behaves unexpectedly always returning today's date.

W1D, 2D3W work as well :-) I am sure there are many more that I missed.

I readily agree that some examples are extreme but what I found quite interesting was that someone put so much effort in making this component so you can type dates in an extraordinary number of ways and most times I find it does what you expect it to.

Monday, June 12, 2006

Finally some new things to do

After a long time dealing with software development pressures at work which consumed my energy to do little else, this weekend I dappled in new technologies (new for me). Feels so good - I have a high from it this Monday morning!

Got introduced to Google Web Toolkit (GWT) which looks very promising. The UI I am ending up with is very clunky and I'll have to work hard on writing (stealing?) some good style-sheets to make the UI look nice. This lead to getting re-introduced to Cascading Style Sheets (CSS) and a whole bunch of related resources. The whole research can be done rather nicely on Google Notebook.

More on GWT as I get more familiar with it.

Saturday, August 27, 2005

A Strange Situation With Buffered Socket Stream

A common technique to buffer socket output stream is to wrap it in a BufferedOutputStream

OutputStream os = new BufferedOutputStream(socket.getOutputStream(), BUF_SIZE);

where BUF_SIZE is the size of the buffer to be used. This works great except in one strange situation where it seems to fail.

Before I get any further, I would like to stress that this piece of information would only be useful to people for who network performance is critical. In applications where it is not, it is best to leave things as they are. Also, this is not related to NIO and might not be an issue in that case. If any one is aware of this please post it in the comments.

Consider an application that writes a small amount of data (say the content length of the data to follow, just a few bytes), followed by larger chunks. (HTTP Chunked encoding is a good example). BufferedOutputStream's write(byte b[], int off, int len) has the following implementation of buffering -


0 if (len >= buf.length) {
1 /* If the request length exceeds the size of the output buffer,
2 flush the output buffer and then write the data directly.
3 In this way buffered streams will cascade harmlessly. */
4 flushBuffer();
5 out.write(b, off, len);
6 return;
7 }
8 if (len > buf.length - count) {
9 flushBuffer();
10 }
11 System.arraycopy(b, off, buf, count, len);
12 count += len;


The if conditions in lines 0 and 8 are for cases where the data being written does not fit into the buffer.

Now, consider our case -
a) Application writes 4 bytes - buffer has 4 bytes.
b) Application writes a larger chunk that exceeds buffer capacity. To prevent needless data copies, BufferedOutputStream flushes its buffer (writing only 4 bytes to the socket stream) and then either copies or writes the chunk based on the conditions in lines 0 and 8.

If this repeats then you see, on the socket stream, writes of size - 4, CHUNK_SIZE, 4, CHUNK_SIZE. (The assumption is that CHUNK_SIZE is atleast BUF_SIZE - 4 for this behaviour) - net effect - the advantage of buffering is lost.

If the application cannot control CHUNK_SIZE, one solution is to override the write in BufferedOutputStream and ensure that a full BUF_SIZE worth of data is written and the remaining (CHUNK_SIZE - BUF_SIZE) is filled in to the buffer.

Monday, May 02, 2005

Reflections On Debugging

This rather pedagogically worded article is a collection of my thoughts on debugging Java software, the programming patterns I have used, some useful API and techniques.

What it is not - it is definitely not complete in terms of information on debugging, its techniques, styles, etc. It is primarily a list of things that have worked for me time and again and a few tools that I keep in my toolkit to use when the situation demands it. I think they will be of use to you as well.

I have been fortunate to work in environments where I touched upon various facets of Java, used various APIs and generally did extremely satisfying work. In all these years, debugging has stood out as an activity that everybody has to perform almost as much as they code or design. I have noticed time and again that being able to debug well is an extremeley useful skill. It can be learnt over time and honed and it is to a large extent the ability to match problem patterns to past issues. People like Rajiv can uncannily pinpoint a problem's cause when they hear its description. This ability comes from years of experience and the intent to learn from every new debugging experience.

Debugging

Debugging is the act of locating and fixing a flaw in software. A flaw can manifest itself in multiple ways. Sometimes it is apparent such as cases when the program crashes or does not do the intended action or does not return the intended result. Sometimes it is hard to say what's wrong when a program does not return or the CPU keeps processing something or when the program does something unexpected in addition to the right action. Debugging, of course, is the action we take post having seen a flaw.

Isolating the problem to code : Identifying where to look for a problem

The problem or flaw appears as a failure of the software to do something it should have. When you encounter a flaw, to debug it, you need to form a mental model of the code to identify where is the code that failed. Debugging largely follows the process of elimination and this process is helped by any symptoms that you can find.

When you have the piece of code that failed you try and find the cause by asking and answering questions - what is occuring? What possible causes could result in this problem? E.g. If something should've happened and it didn't perhaps the code was not reached. Why would the code not be reached? Maybe the if condition under which the method gets called did not evaluate to true or perhaps the if (something != null) check was called when something had the value null.

Another example - If there is an exception then there is additional information about the location of failure and the steps that led to it. The type of Exception will tell you the nature of failure. So if you see a ConcurrentModificationException thrown by an Iterator's next() method you will have to -
a) Find out under what conditions this happens.
b) How could these conditions have been created in your program?
c) Maybe you removed something using list.remove() in your loop, or perhaps you passed reference to the list to some other thread which is modifying it.
Once you have a mental picture of the surroundings of the problem and why it might be occuring, it’s a matter of eliminating the reasons one by one starting from the most likely cause.

Even if you intend to use a debugger, this is a necessary step. You have to always backtrack mentally from the point of failure to locate all possiible causes of failure. A lot of debugging skill relies on this one ability alone.

Reading an Exception

Java Exceptions have a lot of information in them and should be a very well understood to debug problems. Often, I have noticed programmers use the following template for exceptions.
try {
// do stuff here
} catch(Exception ex) {
System.out.println(ex);
}
If you wish to print the exception to know when a problem has occurred then you must consider using ex.printStackTrace(). There are multiple advantages -
A) When using System.out.println(ex), several times there is no message printed other than the class name of the exception that occurred. Imagine having this piece of code in multiple locations, how will you ever know which catch handler printed java.lang.NullPointerException.
B) An exception when printed stands out in a log file or the console. It is several lines long and just the pattern of an exception stack trace print is so different from other message, it is much easier to find than an exception message that looks like other logging statements.
C) Following JDK1.4, chained exceptions get printed as well and you don't have to manage them manually. Root cause gets carried along with the exception.
D) Lastly and most importantly, the stack trace contains a wealth of information that can be used to create a mental picture of what happened.

There have been lots of times when I have looked at a stack trace and said - it should not have come here and been able to trace the problem to a wrong check in an earlier part of the code. You must know how to read an exception. Here's a Java exception printed out.

: Output generated by System.out.println() :
java.lang.ArrayIndexOutOfBoundsException: 0


: Output generated by ex.printStackTrace() :
java.lang.ArrayIndexOutOfBoundsException: 0
at com.sonicsw.tools.test.ThrowException.processArgs(ThrowException.java:32)
at com.sonicsw.tools.test.ThrowException.main(ThrowException.java:21)



1) java.lang.ArrayIndexOutOfBoundsException: 0
The first part of printStackTrace is to do a print out the exception similar to the System.out.println so already you have gotten that for free. This part of the exception stack trace is formatted according to the type of exception and the information printed varies from exception to exception. Some exceptions print nothing more than the class of the exception. Some exceptions (specially custom ones) print a lot of context information that led to this exception.
2) at com.sonicsw.tools.test.ThrowException.processArgs(ThrowException.java:32)
The rest of the exception is the stack trace starting with the location that threw the exception at the top and the caller of the method in which the exception was thrown below it and so on until the executing thread's run method or the main method. The information provided on this line consists of -
the fully qualified class name - com.sonicsw.tools.test.ThrowException
the method - processArgs
the file - ThrowException.java
line number - 32.
Obviously, these are great nuggets of information. In some cases, when compilation does not include debugging information, you can end up with stack traces that do not have line numbers. That’s usually a bummer but atleast you have the stack of methods to locate where the problem occurred.

Other variations for method names are <clinit> for a static initializer - (this is also an example of exception chaining - note the "Caused by:")
java.lang.ExceptionInInitializerError
Caused by: java.lang.IllegalArgumentException
at com.sonicsw.tools.test.ThrowException.<clinit>
(ThrowException.java:21)

<init> for constructors and initializers
java.lang.IllegalArgumentException
at com.sonicsw.tools.test.ThrowException.<init>
(ThrowException.java:20)
at com.sonicsw.tools.test.ThrowException.main(ThrowException.java:24)


and the $number convention for anonymous classes -
java.lang.IllegalArgumentException
at com.sonicsw.tools.test.ThrowException$1.actionPerformed(ThrowException.java:25)
at com.sonicsw.tools.test.ThrowException.
(ThrowException.java:23)
at com.sonicsw.tools.test.ThrowException.main(ThrowException.java:31)


Ofcourse, if the line number is available it makes it a lot easier.

Sometimes, to understand why an exception occurred, you have to understand how control got to that point. The easy exceptions are always the ones which have a localized problem and you can easily catch the problem by the exception. But things can get really hard. E.g. The culprit could lie several methods below and might not even be caused in this thread of execution. A previous event might have generated a bad value which was stored in a field and was picked up by this thread of execution in which the exception occurred. To get to the bottom of these, you need to create a mental back map of methods and events that might have occurred. It always helps to see what methods were called to get to the point of failure.

Trapping Exceptions

Sometimes, something you expected to execute does not occur and there is no information on why it failed. For some reason, control got transferred out of your method. Once you have eliminated any if conditions that are failing it could be an exception getting thrown from somewhere deep inside your code. No exception gets printed though. This usually happens, when you are implementing a piece that fits into a framework. Perhaps, the framework has a logging switch that’s set too low for exceptions to get printed. Or perhaps, the framework is faulty and is not printing exceptions being thrown by overridden methods. Perhaps, the framework is failing because of the unexpected exception being thrown by your method. In such cases, it is best to eliminate this possibility by wrapping the entire method by a try-catch block. For such debugging situations, I prefer to wrap with a try … catch(Throwable t) block because you want to be sure that no Exception or Error is being thrown. Sometimes, when Errors get thrown (such as a NoClassDefFoundError because of a faulty classpath) it will slip through all catch(Exception) blocks.

Debugging is often an exercise in eliminating possibilities and locating the faulty piece of code. This is one technique to achieve that.

When to use Thread.dumpStack

Another neat tool to keep in your toolkit is using Thread.dumpStack() or equivalently new Exception().printStackTrace(). What either of these methods do is to do a printStackTrace() at this line without actually throwing an exception. The usual reason you do this is because you want to know what caused control of execution to come to this point. Problems caused by a method being called unexpectedly can be identified by this.

Using VM Thread dumps and understanding them

My biggest complaint with a lot of experienced Java developers is that they have never heard of the most amazing debugging tool called the VM Thread dump. You can use this technique in innovative ways -
A) To detect deadlocks
B) To diagnose UI hanging problems
C) To diagnose slow UI issues
D) To diagnose spinning/infinite loops
E) For quick and dirty profiling
F) To get an understanding of what the VM is doing at that instant

I cannot do better than this excellent article on this topic available here.

Classpath problems

Another class of problems are classpath issues. There are times when you are not sure if there is another version of a class in the classpath that is getting picked up before yours. Usually a result of a bad environment setup. To eliminate this possibility a simple check is to add a print statement to see if your new code gets picked up. If it isn't getting picked up you need to locate the other class that is getting picked up. One neat API in Java that allows you to locate where a class is being picked up is

Class.getProtectionDomain().getCodeSource().getLocation()

In most cases, depending on the class loader being used you will get the location of the class which is being executed and you can correct your environment setup.

When to use a debugger and when to print

There are times when you should use a debugger and there are times when a print statement is more useful. When you can use what is very well described by the brilliant pieces of work mentioned in the references section. The point I want to make here is that you must know what can be achieved using a debugger. It is an extremely powerful tool and can reduce debugging time quite a bit. It is applicable in many cases but is not suitable for a certain set of problems such as threading issues or issues that show up in long running tasks in which case a log with good print statements is essential.

Good debug printing

Once you have a mental check-list of what the causes might be, you need to eliminate them. Debug prints can tell you quickly if your assumption is right or wrong. Ensure that your debug prints are not causing any side effects inadvertantly. E.g.

System.out.println("value.getCode() = " + value.getCode());
if (value == null)
return;
switch(value.getCode()) {

}

It is important to print sufficient information about the object you are interested in. Perhaps your code is falling through a switch statement without firing any of the case clauses. You will need to print the value of the switch condition. Think a little before you decide to type in any debug print statements. Often when using good debug print statements, the code gets peppered at useful locations with debug statements which can be switched off with a boolean for future use. An easy to use pattern is

private static boolean DEBUG = Boolean.getBoolean("<classname>.debug").booleanValue();

So e.g. if the class is com.test.ArgumentsProcessor then you would write
private static boolean DEBUG = Boolean.getBoolean("com.test.ArgumentsProcessor.debug").booleanValue();

The advantage is that you can switch on debugging for this class without recompiling anything by specifying -Dcom.test.ArgumentsProcessor.debug=true when starting your VM.
E.g. java -Dcom.test.ArgumentsProcessor.debug=true <main-class> <args>

However, the compiler will not remove your debug statements during code optimization in this case.

Using Logging

There is not much that I can add on using Logging that is not already covered by a vast amount of material. You can take a look at the References section below for other information on this subject. The point I would like to make here is that when you are writing print statements that go to a log for debugging purposes, maybe as a patch to a customer to diagnose a particularly tricky problem, think about how the log file might look when it is sent to you. What we think the output would be like when we write log print statements changes dramatically in a live system with multiple threads executing the same log messages. Maybe you need to print the Thread id to bunch all logs or an operation together or perhaps some business data structure id needs to printed with each statement to understand what's happening. You may need to print out the execution path leading to the suspected problem location to know what conditions caused control flow to get there. Another thing to keep in mind is that log files can get so verbose and have so many messages that it becomes very difficult to scan them for problems later.

References

The Art of Programming (Brian Kernighan, Rob Pike) Highly recommended reading
How to be a Programmer (Robert L Read) The first chapter is on debugging and is really well written.

Friday, April 22, 2005

Swing fighting back

Being a veteran Swing programmer and a recent Eclipse plugin developer, I seem to have seen either side of the UI toolkit war.

I have a few interesting observations I would like to share.

Swing Knowledge
We have been trying to hire a few developers with past UI experience (Swing experience is more easily available) but I am apalled at how little the people we have talked to know about Swing. Most people seem to hack in a UI and know just about enough to manage to get the UI working. How a renderer works and what you can and cannot do with it is not understood. Neither did these guys understand that you can write your own TableModel implementation and wire it to your data model and it doesn't have to always be backed by ArrayLists or arrays. I agree that these are not the most knowledgeable Swing programmers out there but when they see a DefaultTableModel they just do whatever they need to to populate it and get their JTable working.

JFace (MV framework over SWT) on the other hand enforces an interface such as a IContentProvider (this has its own limitations though) which needs to be implemented and keeps the model to itself. This is interesting because people then don't think in terms of populating Vectors but consider returning their business objects. I suspect that this might really help in writing better models to their UI. I haven't been able to verify this because of the complete lack of SWT profiles that we have received.

Swing's fight back
I have done a lot of Swing programming in the past and this goes past the regular hack-a-UI. I have been involved in several interesting Swing UIs (IDEs, applications, etc). I have had my share of issues with Swing but over time you learn to work around them. SWT definitely looks good and there are enough articles describing the virtues and problems with either. What is really nice is to see how strongly Swing is fighting back. Several Swing developers are openly defending false accusations (primarily performance related) and there is serious effort at fixing other issues that have remained unfixed over years (probably because there was always something more important to do).
  • NetBeans is being treated by Sun as their flagship Swing product and the competition has really done it (and Swing) a lot of good.
  • Take a look at the new Desktop Java features.
  • Some very important perceived performance fixes such as this.
Swing Performance has been steadily improving and this is an interesting comment on the EclipseWiki
For most of the time that Eclipse was under development, all of the above were very true. It has only been a very recent development (~JDK 1.4) that Swing has gotten fast enough to be acceptable. The look and feel emulations built into Swing are still lacking in terms of platform fidelity (although this is mitigated somewhat by the existence of high-quality 3rd party Swing look and feels).
available here.

All in all I think we'll see very strong improvements in Swing over the next few releases. I am really looking forward to it.

Swing Image
What has still not happened and I am hoping that it will be fixed in the future is that Sun should make sure that any UI they dole out is really good and this also goes for things like Windows Control Panel Java Applet which doesn't look very polished. Sun cannot have any excuses for shipping badly designed UI. Swing's image is affected by such UI and it can potentially undo (to an extent) all the hard work being done.


Check this list of Swing best practises out.


Thursday, September 16, 2004

Usability

Joel has an interesting article on social usability here.
A point he makes (besides the one that the article is actually about) is that

this just goes to show you that usability ain't all that important
But that depends.
Usability is not so important when there is a popular precedent. e.g. Mozilla Firefox doesn't need to reinvent any basic browser UI. They'll add to it and try to make it more usable but no user of theirs has to learn the basics (if they need to, there are enough "browser-gurus" around). Usability is also not that important at a micro-level like in Joel's example. You can go wrong and use buttons instead of tabs and most users won't even notice.

On the other hand, consider ConceptDraw MINDMAP. What they are doing might have a precedent but isn't that well-known. Usability for these guys is their holy grail. And Ramesh will tell you they are great. He just loves the product. He's their power user now. For them, the entire experience is important. Conceptual usability is paramount and then they must have the right micro-level usability worked out too. You see, they are a brainstorming tool. The last thing you want to do in the middle of a brainstorming session is to wonder how you insert another level in the damned diagram. For them, getting it right makes all the difference.

Usability changes over time too. When part of Pramati's Studio initiative, we were amongst the first IDEs that had J2EE development in it. People we showed it to were new to J2EE itself. Especially in terms of using archives like EJB JARs and WARs and EAR files. Over the next release, we had Express Development (tm) (which is present in one way or the other in all J2EE IDEs today) which does away with archiving (deploying source modules automatically - known popularly as exploded deployment) but by the time we had this out, many people had gotten used to the new paradigm. Thankfully we had provided the option of exporting modules to archives (which is what these people used extensively - instantly recognizing the "Export As Archive" option on a module). The point here is that over time, the definition of "good" usabililty had changed.