How does Kaggle rate submissions
Are Kaggle Competitions won by chance?
Kaggle competitions determine the final placements based on a persevering test set.
A sustained test set is a sample; It may not be representative of the population being modeled. Since each submission is like a hypothesis, the algorithm that won the competition may have hit the test set better than the others. In other words, if a different test set were selected and the competition repeated, would the placements stay the same?
For the sponsoring company, this doesn't really matter (likely the top 20 submissions would improve their baseline). Ironically, however, they could use a front-row model that is worse than the other top 5. To the contestants, Kaggle ultimately seems like a game of luck - luck is not required to stumble upon the correct solution, it is necessary to stumble upon the solution that goes with the test set!
Is it possible to change the competition so that all top teams that are statistically indistinguishable win? Or could the most economical or computationally cheapest model win in this group?
Yes, your reasoning is correct. Indeed, if a different test set were selected and the competition repeated, the ranking would change. Consider the following example. All entries to a Kaggle binary label contest are simply guessing randomly (and independently, for example) to predict their output. By chance, one of them will agree to the holdout more than others, even though no prediction is made.
While this is a bit of a contrivance, we can see that deviations in the individual models of the filing mean that applying many such entries, in fact, only matches the noise of the holdout set. This tells us that (depending on each model anomaly) the top N models are likely to generalize the same thing. This is the garden of fork paths, except that the "explorers" are not the same (but it doesn't matter).
Is it possible to change the competition so that all teams win that are statistically indistinguishable from the top performance on the test set?
- One approach (impractical as it is) would be to explicitly identify the variance of a particular model in each entry, which gives us a CI for its holdout performance.
- Another approach that can be computationally high is to bring a CI to holdout performance by making a training and testing API available to all models.
There are other types of competitions where Kaggle has no elements of randomness. For example the stolen sleigh of this Stanta.
It's a discrete optimization problem and there isn't even a private ranking. What you see in the public ranking is the bottom line.
Compared to supervised learning, which is an easy start for many people, this type of competition is "tougher".
- How long is 1 million seconds
- What is climbing
- What do you call frog meat
- What is the base and the threshold
- What are unhealthy types of love
- What are the most comfortable lounge chairs
- Learning the game algorithm is very helpful
- Did affiliate marketing go downhill in 2019
- What are frances cheapest cities
- How much are 1970s records worth
- Is CJ Affiliate Legitimate
- Is it really the best vengeance to live well?
- The Hubble telescope discovered black holes
- Why are water pipes copper
- What kind of coins are valuable
- How many WordPress websites are hacked
- Why are printed circuit boards usually green
- Hallmark offers personalized greeting cards
- What made James Cook famous
- How do I find the median
- Is the life of a PhD student easy
- Kindly cheating is a crime
- Is Quora bad or awful
- What hairstyles did Vikings have
- What Herbalife Products Cause Cancer
- What is fire extinguishing
- Why do people think lawyers are intelligent?
- What is a relational database
- What is an ADX indicator
- Are overweight people considered to be medically disabled
- Still work for NASA
- Who made your day yesterday
- Why were the Vikings so brutal
- Which is better sterling silver or silver