Why is it that some people seem so confident at what they do even clearly their abilities aren't that great? For examples, simply look at failed auditions at talents shows for some cringe-worthy examples:
Likewise, why is that some people who clearly have the ability to do great things, but their confidence is clearly holding him back? For those anime fans, the character Armin Arlert from the anime Attack on Titan is a perfect example.
Confidence is the feeling of competency, the belief that one can tackle challenges based on their skill level. Confidence of course is skill specific; I'm highly confident in my physics knowledge, but totally unconfident in making art and designing buildings.
Ideally how confident a person should accurately match with their skill level. However very often in life, we often see such a mismatch. Over-confidence leads to frustration and disappointment; while under-confidence holds us back to achieve our best.
Why is this so and how can we improve our true confidence without going overboard?
The Dunning-Kruger Effect: The Paradox of Confidence
Suppose you'd like to pick up a new skill, say dancing or cooking. Before hitting the books and trying it, how confident will you feel? Probably totally unconfident, as you know clearly you know next to nothing about that skill, which is why you'll need to start somewhere.
How then do you expect your confidence level to rise as you learn that new skill? Very likely you'll expect it to rise linearly; the more skilled you become, naturally the more confident you become. The following black graph illustrates this expected relationship:
However the reality of how confidence is related to skill level is starkly different. When we just begin learning something and starting to get the hang of it, we have a burst in confidence, feeling like we have already mastered it, giving us a tendency to feel over-confident. This burst of confidence of course does not reflect our true skill level.
Paradoxically, when we start to learn more and get more skilled, our confidence will start to plummet down. At this stage, learning more in fact reduces confidence. This is usually the hard part in learning as you'll start to realise you're not as good as you think you are, and have so much more you don't know.
Only after pushing through this down point, is where you'll slowly gain true confidence. Unfortunately your rise of true confidence will be really rocky; you'll have to go through periodic rise and fall of confidence level, but there will be an overall rise as shown also in the red graph above.
This real relationship between skill and confidence is called the Dunning-Kruger effect, which explains why relatively low skilled people have higher confidence than highly skilled people. In the humorous words of Josh Kaufman in his book Personal MBA, this is why hair-dressers seems to be an expert on everything.
Unfortunately this presents a great challenge when it comes to learning anything, as the fall in confidence usually discourages us to push on and further improve on our abilities, and also gives us an illusion of knowing-it-all when in fact we don't really know everything.
Countering Dunning-Kruger Effect
The good news is that Dunning-Kruger effect can be countered easily: be as objective in your performance as possible.
This is why constantly testing one's own ability is important. Some ways to test own performance can be jumping into doing problems, trying to recall key concepts, simulating test conditions etc. The more objective you are in your performace, the better.
This way, having a clear objective measure of own's performance will allow us to calibrate our own confidence levels. Often there will times where you will have to step up and jump into challenges when feeling unconfident but you know objectively you can do it; and there will be times where you have to tone down your confident levels to avoid looking arrogant.
The bad news is this is easier said than done. Confidence is an emotion that we can't control directly, we can't just say "Yeah I know I have the ability, time to feel more confident!" to ourselves and feel more confident.
Furthermore, constantly testing oneself is a very unpleasant task as it pits our own skill level against challenges, forcing us to see objectively exactly where our skills is. If you're at the starting peak of confidence, this can be a very painful task. Pushing through this pain is necessary for progress.
This is why doing so takes practice. Simply knowing and understanding Dunning-Kruger effect is a big step in making sure our confidence truly reflect our abilities, and the only way to develop real confidence, rather than superficial confidence.
Remember, the only true way to increase real confidence is by actually increasing ability level, and knowing objectively that you can do it. So don't get fooled when others look so confident!