When you follow someone and receive their messages, you'd think that's an action you do, or your server does for you. But it's the other way around: the other account, or its server, is sending these messages to you. It's like being on a mailinglist. You've subscribed once (or clicked 'follow'), but after that they're the ones sending out the messages.

"But can someone put you on their followers list without you clicking 'follow'?" In theory, yes. They can send messages to your server as if you follow them. But your server will not present them in your timeline since the 'follow' is not registered in your account as well. In this case you'll not see these messages. So in practise there's no problem.