Because several hormones in your brain and body take part in the process which makes you horny. Also the brain itself is involved.
But it is surely not because of your sperm. You sperm does exactly nothing. They wait until you release them. Then they swim to the egg and - voilà - a human starts to grow. That's all.
It's like food. You won't surely try to eat if your body wouldn't signal you that it is hungry. And you also won't eat if your tongue wouldn't signal you how good it tastes. It feels nice if you stomach is full. And certain ingredients in the food trigger the release of hormones (I guess e.g. sugar does that).
And about ejaculation: I (tried to) explained that in
my post before: You loose interest afterwards because a) you should be able to fight with lions, bears, wolves and so on or should be able to flee from them. b) Your body also need some recreational time (like
torenz already mentioned in
his post).
And by the way: You're not the only one.