GARMIN eTrex Vista® HCx
NOTE: This GPS unit is now out of date and superceded. It is however a rugged little unit and you could do worse than grab a used one cheap.
What a little ripper! Compact, sturdy, portable, powerful little Handheld GPS unit from GARMIN. The GARMIN eTrex Vista® HCx will get a signal almost anywhere. Has a built-in (barometric) Altimeter and includes an Electronic Compass (which when calibrated are way more accurate than their GPS equivalents) ... plus a raft of other useful and easy to use features.
In it's day the all-up cost was around $390 delivered. You would also need to get a Micro SD card and some high resolution maps to put on the card (using the provided Garmin software).
Update August 2015:
The eTrex Vista® HCx is still working well. Sadly it won't operate properly with the newer Garmin Basecamp software (you will have to stay with the old Trip and Waypoint Manager / MapSource program). I have recently replaced the eTrex with an Oregon 600 - much better ;-)
Of course, without maps a GPS unit is almost useless, so below are some suggestions.
IMPORTANT! - The Australia & New Zealand Base Map included with the eTrex Vista® HCx is pretty ordinary. It is low resolution and not even accurate (it is not the map shown in the picture below). So you will need to get better maps.
There are a couple options for maps which obviously include Garmin's overpriced products. These include Garmin's City Street Navigator and their Topo Mapping kit. At around AUD $200 each, I wouldn't call them great value.
However, if you do decide to buy Garmin map products, I would suggest you also get the nüMaps Lifetime™ Australia & New Zealand product, which provides a life time of quarterly map updates (but read their disclaimer first) for another AUD $190.
Personally, I believe Garmin's pricing and locking maps to a device is a ripp-off. So, I will be using a variety of free maps that can be sourced from:
Open Street Map Australia
Shonky Maps by Shonky Logic
PLEASE NOTE: Maps Shown In Photos Are NOT Included