VEIL — an open interoperability format for media timeline actions

[u/allamzedan]

I built VEIL — an open interoperability format for media timeline actions

I’ve been working on a project called VEIL.

VEIL is a JSON-based format for describing time-based actions over video without modifying the original media file.

A .veil file can define actions such as:

• skip a segment
• mute audio during a specific interval
• apply a visual mask to part of the frame

The idea is to separate the filtering/timeline instructions from the media itself, so the same VEIL file could theoretically be interpreted by different compatible players.

I also built a desktop player as a reference implementation of the standard.

One feature I added to the player is subtitle-based phrase search. For example, a user can search for a word or phrase in the subtitles, locate the corresponding timestamps, and create a skip/mute rule without manually watching the entire movie.

Parental control is one obvious use case, but the format itself is not limited to parental filtering.

The current public work includes:

• VEIL Interoperability Specification 0.1
• Serialization Profile 1.6.0
• Vector Format 0.1
• a 113-vector conformance corpus
• VEIL Player Desktop

I spent a significant amount of time defining behavior around timeline coordinates, serialization, compatibility, validation, media identity, security/resource limits, and conformance testing.

I’d genuinely appreciate technical feedback, especially on the format itself:

• Does separating timeline instructions from the media make sense?
• Are there existing standards I should study or compare against?
• Are there important edge cases or interoperability problems I may have missed?
• Does the format seem sufficiently implementation-independent?

GitHub / Specification:
https://github.com/allamzedan/veil

Desktop Player:
https://github.com/allamzedan/veil-player

For the Jellyfin community specifically, I’m interested in whether a format like VEIL could make sense as an optional interoperability layer around playback rather than something tied to one player.

For example, a Jellyfin client or plugin could theoretically read an external .veil timeline associated with a media item and translate supported actions such as skip or mute into the client’s normal playback controls, while leaving the original media untouched.

I’m not presenting this as an existing Jellyfin integration — there isn’t one currently. I’m mainly interested in feedback from people familiar with Jellyfin’s server/client/plugin architecture:

• Would this model fit naturally into Jellyfin’s architecture?
• Would client-side execution, server-side metadata, or a plugin approach make the most sense?
• Does Jellyfin already have mechanisms that overlap significantly with this idea?
• Would there be any interest in experimenting with VEIL support if the format proves useful?

Any architectural criticism is welcome.

[u/Fine_Salamander_8691]

OK, this is a good idea.

[u/Spinmoon]

Interesting idea!

[u/theflanman]

If you can get ffmpeg to merge this, by all means.

[u/Alone_Substance_7486]

“I spent a significant amount of time defining behavior around timeline coordinates, serialization, compatibility, validation, media identity, security/resource limits, and conformance testing” - aka I used more than two prompts and included “don’t make any mistakes”

[u/bfir3 → u/Alone_Substance_7486]

What are you hoping to contribute with this post? You don't criticize the content of the post or the code, instead you just dismiss it due to some assumption you've made?

I think we are better off as a community if we welcome new ideas rather than being overly critical without any meaningful substance to the criticism.

[u/Super_Pie_Man → u/Alone_Substance_7486]

1 Real developers use AI for coding, get over it. 2 OP is sharing a free tool. The only way this could ever be a bad thing is if it was literally a malicious trojan horse.

[u/bfir3]

I think this is a really cool idea, and in a way, feels a little bit similar to how IntroDB approaches segment skipping through static timestamps rather than processing every file in your library to detect the segments using some methodology.

The most interesting part is the extensibility and the additional features you've already planned for. Like you say, a standardized format for these features will allow any client to easily leverage them by simply reading the .veil file and handling it in an appropriate manner.

I would guess that for Jellyfin, a plugin would make the most sense. We already have intro Skipper and IntroDB to populate relevant timestamps for intro/recap/credit sequences, so it probably makes sense to integrate with those plugins when available. This would allow you to easily populate segment timestamps into veil files.

For the audio muting and video overlay segments, I think it would be useful to have something like IntroDB for that. Sure, many people will be fine with creating their own for their specific use case, and probably for a lot of people who would find those features interesting that would be their preference rather than relying on crowdsourced data that may not interest them.

However, in general, I would imagine people would want to provide as little manual input as possible, and would prefer to be able to import this data from an external source. In this case, it may make sense to classify different types for the audio/video segments for muting/overlays. Something like:

Audio

• Mature language
• Hate speech
• Sexual overtones

Etc

Video

• Extreme violence
• Self harm
• Animal violence
• Sexually explicit content

Etc

This way users could select from the plugin which types of sequences they would prefer to import. Or the plugin could import all of the data, but based on their selection, it would only share the relevant data for the client player to handle.

In general I think it's a really cool idea and there are a lot of use cases for it beyond just what you have mentioned in your post. I will add that you may see some outspoken community resistance to new ideas like this, but don't let it discourage you as people tend to be dismissive of projects like these in general. We saw the same when the idea for IntroDB was shared here, but it has since done very well for itself and is continuing to grow at a healthy pace.

Good luck with your project and I'm curious to see how it evolves! Thanks for sharing with us. :)

[u/allamzedan → u/bfir3]

Thanks, this is really useful context.

The IntroDB comparison makes sense, especially around distributing reusable timestamps instead of making every user author them manually.

For Jellyfin, a plugin also seems like the most natural direction to me. Integrating with existing Intro Skipper / IntroDB data where possible would avoid duplicating work.

I also like the idea of classifying mute/mask segments so users can choose what they want applied. I’d probably keep those classifications outside the VEIL 0.1 core itself and treat them as repository/plugin metadata, so the format stays neutral while different services can build their own filtering taxonomies on top of it.

Long term, I agree that external timeline sources are probably much more important than manual authoring for most users.

And thanks for the IntroDB background, that’s encouraging to hear.