Okay, here's how it is.
Time and space are actually like a bit of fabric. Time goes one way, and space crosses it. The observer travels along the fabric, in a linear path based on paradigms set forth by his species.
(Envision that)
Okay, so you're travelling along the fabric, attempting a straight line. The funny thing is, that it's not a straight line. In fact, it looks something like this:
This happened by accident, Black holes (an as a consequence, wormholes) have such strong gravitational pull, that it exerts force on the entirety of spacetime. (place a rock on your mental fabric)
Because the force is so strong, things going into the holes move faster than the speed of light (which is why light cannot escape). This is a nasty state of affairs, and the extreme amounts of energy burn through the spacetime shirt-thing.
With enough black holes, we end up with a ball of....wibbly-wobbly timey-wimey....stuff. It's all linked together (in places where it touches) and certain points are traversable. This equates to time travel, although it is more exact (and less adjustable) than the romanticized version.