Nvidia has unveiled NitroGen, an innovative AI model engineered to engage in video games directly from unprocessed video sequences, without the need for rewards, game goals, or internal game specifics. Developed by Nvidia, NitroGen processes video frames from gameplay to produce controller actions like joystick movements and button presses. The model undergoes training exclusively through extensive imitation learning, learning by observing human gameplay rather than following specific rules or scores.
NitroGen was trained using approximately 40,000 hours of publicly accessible gameplay videos encompassing over 1,000 games. These videos include on-screen controller overlays, enabling the system to extract player inputs automatically on a frame-by-frame basis. Nvidia confirms that the dataset covers various game genres, emphasizing action, platformer, and action role-playing games.
Fundamentally, NitroGen utilizes a vision transformer for video frame processing, followed by a diffusion-based model that generates gamepad actions over time. The system comprises just under 500 million parameters and performs optimally with games designed for controllers. It is less effective with games heavily reliant on mouse and keyboard inputs, such as real-time strategy or multiplayer online battle arena games.
During testing, the model showcased the ability to execute fundamental tasks across numerous unfamiliar games without prior fine-tuning. Nvidia reported that when adapted to new titles with limited data, NitroGen achieved task success rates up to 52% higher than models trained from scratch under similar circumstances.
Nvidia states that the project serves as a research and development initiative and is not a commercial gaming product. Potential applications include automated game testing, experimental game AI, and broader exploration into general-purpose embodied artificial intelligence, where systems learn to operate in intricate environments solely based on visual input.