Sound quality, first and foremost (and for me, that's usually accuracy and neutrality over "fun factor"), then isolation, then comfort. Then maybe portability.
In terms of the (many) in-ear phones I've tried, I'd say that other than the sound of the phone itself, the next big deciding factor tends to be the wire. The microphonics were the biggest reason I started looking for an alternative to my ER-4P, and the excessive length of that cable made it really inconvenient to use for portable use. The TF10P phones just didn't have good enough sound, and I didn't like the lack of isolation and form factor. Shure has a great product with the SE310 that is fairly comfortable (although their olive-style foamies occasionally irritated my ear, they are very convenient and easy to insert and remove), but I just didn't find that the sound quite fit me, and the wire is a little prone to microphonics, although I'm sure that wire will last for years. The Atrio M5s that I use now have a very supple wire and good memory wires that really do minimize microphonics when worn over-the-ear. Unfortunately, they have only so-so isolation for IEMs, which is why my search continues...