The internal governance structure of a university directly determines its governance ability because a scientific internal governance structure is the key to improve the quality of a modern university.Through a comparative analysis of the shared governance structure and vertical organizational structure of Chinese and American public universities, we find that the public universities in China need to improve their internal governance structure in the following aspects:firstly, to restructure and optimize the internal governance structure so as to focus on the academic affairs; secondly, to redesign the administrative system so that the faculty have more opportunities in academic decision-makings; thirdly, to decentralize powers at the two levels of university and schools based on the matching of work and power.