Restrictive Democracy: Plausible?

Standard democratic practice for decades now has been expansive rather than restrictive.  Democracies attempt to include as many people as possible in democratic procedures.  This is reflected in the enfranchisement of women, racial and ethnic minorities, younger people (via reductions in the minimum voting age, in some places), and in the loosening of restrictions on voting by criminals and the cognitively disabled.  All these moves reflect an underlying acceptance of the idea that (a lack of) competence is the only legitimate reason for not allowing a citizen to participate in democracy.  As more and more people are shown to have (or not clearly to lack) the competence necessary for meaningful political participation, democracies have allowed more and more people to participate.

It seems as though there was at least one other alternative available to democracies.  While I approve of the expansive approach, in which as many people as possible are allowed to participate, it is not clear to me that a restrictive interpretation of a capacity requirement was impossible.  What do I mean by a restrictive interpretation of a capacity requirement?  I mean that we could hold potential democratic participants to a higher standard than we currently do.  The capacity requirement as currently used is minimal.  We do not require that our citizens are good democratic citizens.  We do not test citizens for political knowledge, or make any attempt to ensure that when they cast their votes they do so on reasoned grounds.  For our current purposes, the vote of someone who spends every available minute determining which political party or candidate would best represent their interests, is as valuable as the vote of the person who chooses who to vote for by drawing a name from a hat. 

We could, it seems, have reasonably chosen to interpret the capacity requirment more strongly, more restrictively.To do so, we would have to claim that there are benefits arising from having capable* democratic citizens (that is, strongly capable citizens) that do not arise from having minimally capable democratic citizens.  What could these benefits be?  A reduction in votes randomly assigned, a reduction in votes assigned for sub-optimal reasons, potentially a higher turnout amongst capable* citizens, as they internalise the idea that the right to participate in politics is now a privilege earned through demonstrated ability.  We may also want to claim that there are benefits to democracy as a whole.  Epistemic benefits arising from having more capable voters on average would be a primary example.

What would we lose?  Impartiality seems to be the big one.  Some people would have advantages over others, whether through greater innate ability, or (far more likely) through privileged access to an environment that encouraged learning.  This access would arise through being part of the dominant cultural group within society, by having money, in all likelihood (particularly amongst the poorer and minority groups within society) by being male, for gender imbalance is often more pronounced in groups who are already least well off.  Further, if we wanted to run a line like this, we would have a strong incentive to apply impartiality in the use of the capacity* requirement.  A minimum age limit would no longer be reasonable (if it is at this time), as we have an explicit capacity* requirement which we can use to exclude the incapable.  Why wouldn't the capable* young be included?