I've explained this several times over an IM conversation and I'm just getting sick of doing it. So I'll continue to clean this up to try to make it easiesr on myself to explain to you guys 
I'll do my best to explain using a D16Y8 2nd gear as example.
Here is a picture of an assembled d16y8 countershaft (thanks Jason!)

I'm going to try to keep from getting too technical here, so someone might read this and be like (you left a bunch of stuff out. Yes, Mista Bone I mean you
)
okay, here we go:
The following items are locked onto the countershaft and they don't spin:
synchro hub
synchro sleeve
the synchro sleeve can freely move a slight amount (see the brass tab that is between the synchro hub teeth)
2nd gear spins at high rpms on the countershaft.
When you are engaging 2nd gear the synchro sleeve slides down towards 2nd gear. When this occurs the brass synchro bites down onto the gear cone (pictures below) and it acts like a brake that stops 2nd gear from spinning. That's it. All it is is a brake pad for your gears!
After many many gear shifts, the synchro teeth begin to wear out. Just like a brake pad would. Eventually they will wear out so much that it will be the dog teeth hitting the teeth on the synchro sleeve. This is what happens when gears "GRIND"
So what happens when the gears grind long enough? The dog teeth start to wear out and the sleeve is no longer able to fully engage the gear. When this happens, your gears begin popping out of gear.
synchro sleeve (note the dog teeth on the inside)

New synchro, notice the ribbed teeth that bite down onto the gear cone.

you can see the gear cone here, also notice how worn out the dog teeth are.

A way of measuring the life of the synchros (92-95 D trans)

Ignore these pics for now, I'll elaborate on gear cone scuffing later 
http://i107.photobucket.com/al...5.jpg
http://i107.photobucket.com/al...7.jpg