You can hide the Twitter Follow button only by adding the following lines to the panel.ini configuration file.

To hide the Follow button:

[twitter]
showFollowLink = false

To undo the change and return to the default values, remove these lines from panel.ini.