The fabric on the driver side door is starting to fall off the panel so my thought was to take off the arm rest off the door and fix the fabric but I can't seem to get the armrest off the door itself.

I can only find one screw going into the top of the armrest, there are no other screws holding the actual arm rest.

Do I actually have to remove the entire interior door panel to get the armrest off the panel?

 

You have to remove the door panel to remove the screws of the armrest

 

Thank you very much.

Took the panel off and like you said the screws and clips that hold the armrest are on the backside of the panel.