You can lose a lot of heat through your chimney, even with the damper closed. Installing fireplace doors is a great way to eliminate that draft. They're easy to install. Most models attach by tension, with simple hardware that doesn't require drilling or damaging your fireplace brick.
For more on fireplaces, consider:
Hot Fireplace Inserts
Fireplace Maintenance Checklist
Quick Tip: Make Your Fireplace More Efficient