NiceShot AI – The analytics layer competitive games forgot to add

I've been working on NiceShot AI for the last 26 months as a way to experiment with computer vision for gameplay analysis.

The basic idea is to take a recorded long gameplay session and automatically turn it into structured gameplay information and highlights. Bridging the performance analysis gap between match stats (too short) & lifetime stats (too long).

The current pipeline is:

Gameplay video → YOLO detection → event tracking → OCR/context filtering → event timestamps → clips → highlight compilation → session statistics

For example, I can fine-tune a YOLO model to recognize game-specific HUD elements and then configure the pipeline to interpret those detections as events such as kills, deaths, or medals.

One thing I wanted to avoid was making the entire system game-specific. Supporting another game should mostly require a new detector/model and configuration rather than rewriting the whole pipeline. And currently, I am working on that.

I also use OCR for states that shouldn't be counted as gameplay events. For example, in Call of Duty games a kill indicator can appear during a "KillCam" or "Spectating" state, so the OCR layer can be used to filter those cases.

The system can then produce clips around detected events and compile them into highlight reels, including vertical versions for Shorts/TikTok-style content.

Recently, I have also been experimenting with using a lightweight Video LLM after event detection. Instead of sending the entire multi-hour gameplay session to the Video LLM model, I take the extracted short clips around an interesting event and ask the model to explain what happened. The goal is to eventually turn this into a lightweight conversational gameplay coach.

The project currently runs locally on NVIDIA GPUs. Have experimented with an old GTX1650 4GB VRAM laptop and it works fine but slow ofcourse.

Finally, I know that real-time event detection should be the way to go and this was my original goal, but I chose offline processing because I didn't want computer vision inference to interfere with the player's actual game performance.

I'd also be interested in hearing how others would approach making the detector/inference layer fast enough for real-time gameplay without causing noticeable GPU/CPU overhead. Keeping in mind, I input 1080p frames into the model.

The project is still evolving, so technical criticism and suggestions are very welcome.

Thanks.