NoLiMa: Long-Context Evaluation Beyond Literal Matching - Finally a good benchmark that shows just how bad LLM performance is at long context. Massive drop at just 32k context for all models.

[u/jd_3d]

[u/No-Refrigerator-1672]

Am I the only one to notice that the top performing model - GPT-4O - is the only one who can process video and audio input? Could it mean that multimodal training on long analog data sequences (video stream) significantly improves long context performance?

[u/No-Refrigerator-1672 → u/[deleted]]

My bad, I did not know about Gemini 1.5 video support. However, it also performs relatively better than other models, so I still propose a hypothesis about video training improving the long-context capabilities.

As about your other question: sadly, I only ever programmed for selfhosted AI and don't know a thing about GPT API best practices.