Also:

• Remember last used directory (I hate all progs which always returns to My Documents or its install dir)
• OK and Cancel buttons (or other similar ones) should be bound to Enter and Esc keys
• App should have working uninstaller (especially for larger apps)