Where to buy
Buying a bike from a reputable local cycle shop usually means you get good advice & a lifetime frame warranty (although often shorter for carbon). They should also perform regular free service checks.
Frame
A light alloy frame will provide strength and durability whilst being light enough not to slow you down.
Brakes
Disc brakes are a must and the small extra cost for hydraulic brakes is definitely worthwhile.
Suspension
Depending on budget, front suspension is s must have. It's worth getting better quality front forks before considering rear suspension.
Wheel size
26 inch wheels are easier to manouvere
29 inch wheels are easier to ride over objects
27.5 inch wheels are a good compromise
Pedals
Clip in shoes will provide better power transfer, allowing to pull the pedals as well as pushing them. To start with, make sure the clips are easier to clip out of, but be aware that you'll probably fall over with your shoes still clipped on to the pedals.
Tyres
If you're going up hills make sure you've got tyres that have good grip.
Rough price guide (new bikes)
$500 - very basic entry level, front suspension
$1,000 - good hard tail with hydraulic disc brakes and decent front forks
$2,000 - entry level full suspension
$3,000 - good full suspension
$4,000 - $10,000 - better gear & lighter frames, dropper posts, lock out front & rear air shocks
Extras
Multi tool
Puncture repair kit
Spare tube
Small pack with water bladder
Snacks
MTB clothing
Lights
Decent helmet
First aid kit
GPS
Camera
Car rack