There is only one realistic answer to this question: we dont know. Nevertheless, historically several theories of how society/sexuality and gendered behaviour should be organised have been based on the assumption that the answer to the question is known to the moral organisers of society. As a very great generalisation, in Christian countries, which have seen some forms of sexual expression as wrong:
a) if you think that men cannot control themselves sexually (at least some of the time), then you emphasise marriage (as a legitimate outlet for male sexuality) and segregation of the unmarried. (One version of this is male monasticism, another is severe restrictions on the movements of women). In this view, (Catholic) priests, who must be celibate but are not cloistered, are almost bound to fall into sin (a common view ever since Protestantism got going). (Interestingly, you get the same patterns if you start from the medieval view that it is women who are unable to control their lust).
b) if you think only a spiritual elite of men can control their lust, then priests as well as monks can possibly manage OK. Laymen are then definitely an inferior group, ranked in their weak will with bad priests. (In both a) and b) of course, women, or at least any who exhibit tempting behaviour, by being sexually desirable in any way, are automatically worse).
c) if you believe that most men can potentially control their lust, then laymen and women arent automatically at the bottom of the heap. Instead of misogyny, there is more of an emphasis on male responsibility for controlling their own lust, as a conquerable sin. (I would argue this is the Carolingian position).
What is odd is that these three very different views, all of which have been held and are still held today, are based on the answer to an intrinsically unanswerable question. I cant think of any way, even with modern research methods, that you could study this question accurately. The chances of getting accurate answers from surveys, however well defined, seem to me low for such a sensitive subject and there are few experiments that would avoid bias. There are some (fairly gruesome) methods for measuring sexual desire in men, but they dont tell you anything about the interaction of body and will. Now consider how you would go about getting an answer in an age before social science. You are reliant either on dodgy medical theories about the genitals and observation/discussion. Jerome was happy to use anecdotal evidence for the unstoppable power of the sexual drive (such as the eighty-year-old bishop who suddenly married), but his primary reason for seeing sex in this way was his own experience. Jerome makes explicit that his lust tormented him even as an ascetic in the desert. The same is probably true for the many medieval clerics and monks who spoke of the overwhelming force of lust in generalised terms: it is their own experience that they are reflecting. Who else would they be willing to ask and might they obtain a truthful answer from (excluding possibly their most intimate friends)? How much has the history of Christian discussion of sexuality been influenced by the sexual sensitivities of a handful of men?