The muscles in singing that determine vowel definition are primarily the arytenoid muscle system, and secondarily the oro-pharynx – the back of the mouth-throat area.
In an ideal situation:
This is the ideal process, complex to be sure. The point to note is that the vowel first originates as a thought in the mind, and initial formation is at the level of the cords with subsequent formation in the oro-pharynx. Then, the center and front of the mouth, next the tongue, and finally the lips and teeth are important in this process. There should not be an overt effort to control these subsequent shapers and finishers of the vowel.
Note that sometimes the position of the mouth or tongue or lips might need to be addressed so as to help in correcting faulty vowel formation. However, if the vocal cords are not adjusting properly for pitch with a concomitant vowel size and shape at the level of the cords and oro-pharynx, then nothing one does with the front of mouth, tongue, or lips will help accurately define the vowel.