usually happens when im at a complete stop, but goes away when i press the gas. what could it be.

timing belt maybe?

A/C tension pulley. Notorious for squeaking when they get older.

If it's a whistling sound, maybe a vacuum leak? That would go away when you open the throttle.

The MOST important thing is listen & narrow down the location of the noise.

Does this require a new one, or just a tighten of the belt?

how much are those tension pully?