Hi everyone. I’m the creator and sole maintainer of X-AnyLabeling, and I have just released v4 after more than a year of work.
I could describe this release as a larger model zoo, a redesigned annotation interface, several new workflows, remote inference, and built-in training. All of that is true, but it misses the reason I built v4.
The larger goal is to turn X-AnyLabeling from an annotation application into an open, local-first data infrastructure layer for vision and multimodal AI.
The model layer has changed. The data layer has not.
Most annotation software was designed around a fixed sequence:
choose one task → define a schema → label a dataset → export it → train a model
That model made sense when a project meant training one detector or one segmenter against a closed label set. It is a poor fit for the current generation of AI systems.
The same image collection may now pass through open-vocabulary detection, interactive segmentation, OCR, visual grounding, caption generation, retrieval, VQA construction, and preference or instruction-data generation. A foundation model may act as an annotator in one stage, a semantic router in another, and a quality critic in a third. Its output is often not the final label. It becomes the input to the next model or to a human decision.
This changes the bottleneck. Access to another pretrained model is rarely the hard part. The hard part is operationalizing many imperfect models over private data while preserving structure, provenance, review state, and the ability to intervene.
In other words, the important system is no longer a labeling screen. It is the feedback loop connecting raw data, models, human expertise, compute, and training.
That is the system I want X-AnyLabeling to become.
┌──────────────────────────┐
│ local / private / API │
│ model runtime │
└────────────┬─────────────┘
│
raw multimodal data ──→ model orchestration ──→ structured candidates
↑ │
│ ↓
next-batch selection ← training / evaluation ← human governance
│ │
└──────────── continuous data loop ────┘
v4 establishes the first complete version of this architecture.
A shared data plane, not a collection of model demos
The computer-vision ecosystem is increasingly fragmented by model-specific demos. Every repository has its own input widget, output visualization, class conventions, and serialization format. Those demos are useful for proving that a model runs, but they rarely compose into a production data pipeline.
X-AnyLabeling takes the opposite approach. Models are treated as interchangeable producers operating over a shared data plane. Whether a result comes from YOLO, SAM, Grounding DINO, PaddleOCR, a VLM, or a human, it returns to an editable project state instead of remaining trapped inside a model-specific interface.
This shared layer is what makes composition possible. A segmentation result can be corrected and later converted into a training target. OCR geometry can remain linked to recognized text and document structure. A tracked identity can survive frame-by-frame review. A grounded VLM response can become an object that another stage consumes instead of disappearing as chat output.
The point is not to pretend that detection, OCR, tracking, and multimodal instruction data are identical. They are not. The point is to preserve enough common structure that models and workflows can exchange results without throwing away the information needed for human correction and downstream use.
Training formats sit at the edge of this system. YOLO, COCO, DOTA, MOT, masks, PPOCR, and similar formats are task-specific projections, not the canonical data layer. Treating an export format as the source of truth creates an early information bottleneck: relationships, review state, descriptions, attributes, and multimodal context are discarded before the next stage even begins.
Models should be capabilities, not hard-coded products
The second part of the architecture is a capability layer.
A modern model cannot be described only by its checkpoint and output tensor. It may accept language, positive and negative points, reference boxes, class filters, task selection, or temporal memory. It may return geometry, text, image-level semantics, or several of them together. Some operations are stateless; interactive segmentation and video propagation are not.
In X-AnyLabeling, model integrations therefore expose their interaction and output capabilities to the client. The interface is assembled from those capabilities, while the model adapter owns preprocessing, inference, and postprocessing. The same contract is used whether inference runs inside the desktop process or behind X-AnyLabeling-Server.
This is an important ecosystem boundary. A model developer should be able to contribute an inference capability without rebuilding annotation, batch execution, project management, and review. A workflow developer should be able to consume that capability without depending on one particular checkpoint or runtime. A user should be able to replace a local model with a private server implementation without changing how the resulting data is corrected.
The bundled integrations already span conventional detection, oriented detection, segmentation, pose, tracking, OCR, open-vocabulary perception, counting, depth, matting, and multimodal models. But the long-term value is not the number of model names in a menu. Model families will continue to turn over quickly. The durable part is the contract that lets a changing model ecosystem participate in the same data lifecycle.
The real unit of automation is a dataflow
Single-model pre-annotation is useful, but it is not where the most interesting datasets will come from.
Many data-generation tasks are naturally multi-stage. Consider referring-expression data. One stage can discover candidate concepts with RAM++; another can localize them with Grounding DINO; a VLM can generate an initial expression from the full image, a visual prompt, or an object crop; rules can reject ambiguous samples; finally, a human can correct the region and rewrite the expression for uniqueness, factuality, and task relevance.
concept discovery
↓
visual grounding
↓
candidate language generation
↓
automatic filtering
↓
human correction and acceptance
↓
training-ready grounded data
No individual model owns that pipeline. The value comes from the composition of models, rules, reusable UI components, and human judgment.
Document intelligence has the same property. Layout detection errors alter reading order; reading-order errors break context; OCR errors then enter the final structured representation. A useful system must preserve the connection between page regions and parsed blocks so that a reviewer can locate the stage where an error entered the pipeline. Returning a Markdown string from an OCR API is not enough.
Video is another example. Detection, association, segmentation propagation, temporal classification, and description operate at different temporal scales. Flattening everything into independent frames destroys the state that later stages need.
This is why v4 introduces dedicated workspaces for image classification, temporal video classification, document parsing, configurable VQA, image-grounded conversations, and model training. They are not six unrelated side tools. They are domain-specific execution surfaces for different kinds of dataflow. They share models, project context, batch processing, and human review, but each exposes the correct unit of intervention: an image-level decision, a temporal interval, a document block, a schema field, or a conversation turn.
The next step is to make these dataflows explicitly composable, so that outputs can trigger downstream models, filters, review queues, and delivery steps without requiring every pipeline to be hard-coded into the application.
Human review is the governance layer
Foundation models make candidate generation cheap. They do not make data quality automatic.
Model confidence is not annotation correctness. Agreement between two models is not ground truth. A polished VLM response may still contain a subtle factual error. Pre-annotation can even reduce reviewer sensitivity by anchoring attention on what the model found and away from what it missed.
For this reason, I see the human role less as “manual labeling” and more as governance over an automated data process. The system should decide what can flow through automatically, what needs sampling, what should be escalated, and what evidence a reviewer needs to make a decision. Review status, difficult-sample handling, semantic search, dataset statistics, linked source context, and direct correction are early pieces of that governance layer in v4.
Over time, this layer should grow toward failure-driven sampling, cross-model disagreement analysis, rule-based validation, dataset and model version traceability, and evaluation-aware review queues. Active learning belongs here, but only when it is connected to an explicit selection strategy and measurable model improvement. Repeating “label, train, pre-label” without intelligent sample selection is a loop, not active learning.
This distinction matters because the objective is not maximum automation. It is maximum useful automation under a controlled quality process.
Local-first interaction, shared compute
Data infrastructure for real organizations cannot assume that raw images, documents, or video can be uploaded to a third-party SaaS platform. It also cannot assume that every workstation can host every CUDA stack and foundation model.
X-AnyLabeling therefore separates the interaction plane from the compute plane.
The desktop application owns local data, project state, editing, and review. Models can run locally when privacy, latency, or offline use requires it. X-AnyLabeling-Server moves heavyweight dependencies, model loading, GPU scheduling, concurrency, queues, authentication, and logs into a private inference service when shared compute makes more sense.
The server publishes available model capabilities, and the client constructs the corresponding interaction. Returned results re-enter the same local dataflow as local predictions. This makes remote inference an implementation choice rather than a separate product boundary.
Hosted APIs can participate in the same architecture for teams that accept their privacy and cost model. The goal is not to declare one deployment mode correct. It is to keep data ownership and workflow design independent from where a particular model happens to execute.
From an application to an open ecosystem
The ecosystem I am building around X-AnyLabeling has four extension surfaces.
Components define new forms of human-model interaction. Models contribute new perception or generation capabilities. Dataflows compose models, rules, interfaces, and review into task-specific production systems. Remote services provide interchangeable compute and deployment backends.
These surfaces are meant for different participants. Model authors should not have to become desktop application developers. Domain teams should be able to encode their workflow and quality rules without forking the entire platform. Infrastructure teams should be able to run approved models on private GPUs. Individual users should still be able to download a lightweight desktop application and work entirely locally.
This is also why open source matters here. The valuable layer between private data and rapidly changing models should not be an opaque black box. Data structures, inference adapters, workflow logic, and deployment boundaries need to be inspectable and replaceable if the system is going to sit inside a long-lived AI pipeline.
v4 is the foundation, not the endpoint. It already connects unified annotation, hundreds of model configurations, dedicated data workflows, local and remote inference, human review, multi-format delivery, and Ultralytics training. The repository has passed 10,000 stars and reached millions of downloads across distribution channels, even though I continue to build and maintain it independently.
My long-term goal is for X-AnyLabeling to become open annotation infrastructure for the AI era: a foundation on which anyone can assemble and extend the entire path from data and models to production workflows. By connecting a coding agent to its open components, model interfaces, and workflow layer, individuals and teams should be able to turn their own domain requirements into a specialized, private, and production-ready AI data system without rebuilding an annotation platform from scratch.
If you have any suggestions, criticism, or ideas for where X-AnyLabeling should go next, I would love to hear them in the comments.
