Vision in chatbots enabled reasoning over diagrams, screenshots, and scanned documents in one thread.
In 2023, OpenAI enabled vision in GPT-4 (GPT-4V), allowing image uploads in chat. Google, Anthropic, and others followed with their own multimodal stacks.
Use cases: accessibility, STEM tutoring with diagrams, UI analysis. Also new visual adversarial attacks and privacy leaks via photos.
The frontier shifted from “text in–text out” to agents that see screens and act—a precursor of computer use.
OpenAI launched GPT-4 with notable gains on professional exams, robustness, and later visual ability.
Frontier systems combined diagrams, PDF papers, and code in one tool-verified reasoning thread.
OpenAI presented CLIP (text–image contrast) and DALL·E (text-to-image generation), catalyzing the multimodal wave.