I did a quick search, but with no relevant results... But we have a 07 CR-V and noticed that the headlights quickly dim when the fan kicks in. Is this a sign that the alternator is about to go bad?

Car is completely stock and the only thing that is different are interior bulbs have been switched out to LED.

Battery was replaced a couple of years ago with an Autozone battery, which I'm now thinking might be culprit. Just wanted a second opinion though...

 

go to your local autoparts store. they should have a battery/alternator tester. Alternatively you can use a multi meter. make sure the car hasnt been running in awhile (2 hours or so) and check the DC voltage (on 20 mode) of the battery. It should be mid 12V (12.3-12.6). If its lower, it indicates the battery is on its way out. 11 volts is considered dead in automotive. A car might still start on 11 volt, but itll take the alternator out as it will always be working a full tilt. Next start the vehicle and turn off everything inside the vehicle (radio, fan, heated seats, ect). check battery voltage. Should be 14.7. Turn everything inside the vehicle on (radio, fan, heated seats, ect) and check voltage again. It should still be mid 14's. If it isnt, it indicates your alternator might be on its way out. It is common for both a battery and an alternator to be shot if left unattended. Should you find that one or both need to be changed, bring it to a professional or the auto parts store(so they can test it) for a second opinion.

 

agreed, check the voltage. just sounds like a common honda problem to me though.