Can't imagine why you'd want to ask your ghost followers to leave. They may not comment or like as they ought to, but they may still view your paintings and enjoy what you do.
I have like 650 ghost followers (rough estimate) but I don't mind, because they do still view my handiwork. I don't want to be famous or popular, but I do like the thought that people enjoy my art work, even if they do remain silent about it.

And besides, you've been a bit of a ghost follower to me as well. :3 But I won't ask you to unfollow me, if you enjoy at least some of the stuff I do. Just saying.