GPT-4 (2023) › Limitations

GPT-4 was costly to train and, for high-volume applications, costly to serve through an API. OpenAI did not disclose the full training compute, but access to comparable development resources was limited to well-funded organizations.

Biases in web and corpus data could appear in GPT-4's outputs and affect downstream applications. Mitigation required targeted evaluation, changes to data or post-training, and monitoring in the intended deployment context.

GPT-4 still failed on logical deduction, symbolic manipulation, and multi-step problems. A detailed explanation could contain an invalid intermediate step even when its prose sounded confident.

Image input also had failure modes. Small details, spatial relations, unusual diagrams, and ambiguous images could produce incorrect descriptions or inferences.

The context window bounded how much text or conversation history a request could include. Applications handling longer material had to split, retrieve, or summarize content, each of which could lose relevant information.

Safety mitigations reduced some harmful behavior but did not prevent every unsafe, biased, or factually incorrect output. They could also cause false refusals on benign requests.

Performance varied by domain and prompt; results also depended on the evaluation method. Strong exam scores therefore could not substitute for testing the model on the actual distribution and failure costs of an application.

The model also offered little traceability from a response back to training examples or internal computations. This complicated debugging and auditing. It also constrained use in settings that required an accountable explanation.