Just IMHO; despite the FFM-Threesome antics and Alcohol-kun working as an aphrodisiac on many of the ladies, MC is still portrayed as (relatively-speaking) rather indecisive, if not a total pushover ... not sure whether artist-san set him as a placeholder for the reader to self-insert.
As for the girl who walked away, it seems that the Maid/Nee-San just might know something. On that basis, I'll come back for the next chapter.