You are looking at the topic of slavery from the wrong perspective. Slavery is not brought on from racism, but rather racism is a symptom evolved from slavery. In order for people to come to grips with the concept of owning another person, it is necessary to dehumanize the one you choose to enslave. After that, it becomes easier to justify your actions and continue them.
When the Confederate slaves were finally emancipated, the former slave owners still felt that African people were inferior and could not relate to them as equals. Even many of those, who supposedly advocated the abolition of slavery, still felt they were not quite equal. It is that false feeling of superiority that has evolved into the concept you are referring to as
racism.
As to your question:
Is slavery bad. Is owning one man bad? Or is it just one group of men. And does it make a differenct if there treated well or not?
Yes, slavery is bad. It is an act that strips an individual of their right to choose their own path in life. It tears away a person's free will and self-respect, and replaces them with blind obedience and self-loathing. While there are some people who relish being submissive, and like being controlled, I doubt they understand the harsh reality of true slavery. Being treated well, as a slave, comes from being obedient and not from the kindness of the owner. Good work is rewarded and bad work is punished severely! That is the truth of slavery, and any other belief is a fiction created by those who enjoy living in fantasy land. The true scars come when the slave begins to believe the lies their master's tell them. Those scars are very difficult to heal.
What we see in S&M or BDSM films is not real slavery, but sexual role play. Most people will find little to arouse them if they saw an actual slave staked out and whipped until half dead, and then being forced to continue working in the fields.