I would check four things:
First thing: Use a lot of Arbiters:
Nodes do not vote for themselfs. So you should run Arbiters on client side to represent their view on the quality of the ReplicaSet members.
If you have package loss between a secondary and the master the secondary will not vote...