Researchers from KAIST have developed a novel technique that trains neural networks to recognize unfamiliar information and lower their confidence levels in uncertain situations.
The study, led by Professor Se-Bum Paik, reveals that the root of the issue lies in the “random initialization” method commonly used in deep learning models. Neural networks often display high confidence even when processing random signals before they begin working with real data. This initial bias persists during training, leading to hallucinations in generative AI.
“By incorporating key principles of brain development, AI can recognize its own knowledge state similarly to humans. This is crucial as it helps AI understand when it is uncertain or may be wrong, rather than just improving the frequency of correct answers,” says Professor Paik.
The solution draws inspiration from biology: the human brain forms neural connections through spontaneous activity even before birth, in the absence of external stimuli. Researchers adapted this experience to the digital realm by introducing a warm-up phase.
- Training on Noise: Before engaging with real objects, the algorithm is briefly trained on a random dataset devoid of meaning.
- Adjusting Uncertainty: During this phase, AI minimizes its confidence level, effectively learning the state of “I currently know nothing.”
- Base Adjustment: After this preparation, the model begins training from scratch, where its confidence in responses is directly proportional to the quality of the knowledge acquired.
The application of this new strategy has significantly improved AI’s ability to recognize data that differs from the training sample, known as out-of-distribution detection. In scenarios where conventional models would confidently err, the updated algorithm indicates a high level of uncertainty.
“This is essentially a step toward creating metacognition in machines—the ability to distinguish between ‘what I know’ and ‘what I do not know,'” the researchers assert.
The scientists emphasize that this approach not only enhances the accuracy of responses but also makes AI behavior more predictable. They believe the technology can be applied to any deep learning architecture, thereby increasing the overall reliability of artificial intelligence systems worldwide.
KAIST researchers have introduced a method to improve AI's ability to handle unfamiliar information by lowering confidence levels in uncertain situations. This advancement aims to enhance the reliability and predictability of AI systems globally.