(Note: I don't consider curls loosened by heat tools "damage". Below, I'm talking about actual damage to your strands such as split ends etc. If you're talking about a loosening of a curl pattern, that's something different IMO.)
In my experience, using a quality flat iron with ceramic plates that emit even heat will give you less damage than using a blow dryer. Blow drying is a lot of manipulation on hair in its weakest state (wet), plus it tends to dry out the hair and rough up the cuticle.
I've seen a ton of long, healthy-haired beauties who either airdry or rollerset and flat iron weekly. I don't see many long, healthy-haired beauties who blow dry weekly and do some sort of stretched style.
YMMV however. I find that I can do either or (either blowdry or flat iron) regularly (for me roughly every other week) and still retain length/healthy hair.
Ultimately you have to know/learn your hair, and know/learn how to use heat tools to avoid damaging your strands. Anything can cause damage if you don't know what you're doing.