How reliable is the ranking?
We test the ranking on simulated projects where we know which ideas are really best. Here is what we found, including where it falls short.
Does it find the best ideas?
Mostly, and it gets better with more people. In each test every idea has a known true quality, and simulated people usually, but not always, pick the better idea in a pair.
| Project | Ideas | People | Comparisons per idea | Did the best ideas make the top 5? | Is the whole list in the right order? |
|---|---|---|---|---|---|
| Small team25 ideas · 30 people · 24 comparisons per idea | 25 | 30 | 24 | 77% | 0.90 |
| Department45 ideas · 75 people · 33 comparisons per idea | 45 | 75 | 33 | 76% | 0.93 |
| Large department45 ideas · 150 people · 67 comparisons per idea | 45 | 150 | 67 | 83% | 0.96 |
| Company-wide45 ideas · 400 people · 178 comparisons per idea | 45 | 400 | 178 | 87% | 0.98 |
| Company-wide, many ideas120 ideas · 400 people · 67 comparisons per idea | 120 | 400 | 67 | 78% | 0.97 |
Each test has a known true top five. The first result is the share of those five that appear in the ranking’s top five. The second compares the whole ranked list with the true order: 1.00 is a perfect match, and the closer to 1.00, the closer the list.
In plain terms: the overall order is very reliable, and the exact top five is right about four times in five. Ideas near the cut-off can swap places, so treat the top few as a group rather than a strict order. How clearly people agree matters too:
| How people vote | Did the best ideas make the top 5? |
|---|---|
| Voters often disagree about quality | 70% |
| Typical | 80% |
| Voters mostly agree about quality | 88% |
Does the order votes arrive in change the result?
A little. The ranking uses a rating system that updates after each vote, so the same votes in a different order can nudge it. We replayed the same votes in random orders: the top five stayed the same 91% of the time, and the overall order matched at 0.99.
We also ranked the same votes with a second, order-independent method (Bradley-Terry). It found the best five 82% of the time, against 81% for ours, and the two agreed on the top five 94% of the time. We did not find a reason to switch, and we will keep testing as we change things.
Can an organised group push an ordinary idea up?
Not easily. We simulated a group who always vote for one ordinary, middle-ranked idea whenever it appears. Each person is still limited to the same number of votes as everyone else.
| Share of voters pushing it | Typical finish (of 45) | Reached top 10 |
|---|---|---|
| NoneNo one pushing it | 23 | 0% |
| 5%5% of voters pushing it | 22 | 0% |
| 10%10% of voters pushing it | 22 | 1% |
| 20%20% of voters pushing it | 20 | 1% |
| 30%30% of voters pushing it | 17 | 10% |
It took around a third of all voters acting together to get an ordinary idea into the top ten, and even then not often. Larger groups do move it, so very close races should be read with care.
What these tests do not show
- They are simulations. Real people are less predictable. We use them to catch design faults, not to promise results.
- One person with many identities. These tests give each simulated person one vote allowance. Protection against someone voting from many links relies on vote and device limits, which we are testing separately.
- Quality is not the same as value. The ranking shows what a group prefers. It does not know cost, risk or feasibility. That is what the improve and decide steps are for.
- Very small groups. With few people the order is noisy. We recommend 25 or more.
Help us check it
If you work with statistics, decision science or survey design and see something we should test, tell us. We publish what we change and why. Use the contact form.