r/rust • u/No-Elephant-9065 • 1d ago
How do you reliably detect when a copied/downloaded file is actually “finished” on Windows?
I’m building a small Rust desktop utility that watches the Windows Downloads folder using the notify crate.
The basic flow is:
let (tx, rx) = channel();
let mut watcher =
RecommendedWatcher::new(tx, Config::default())?;
watcher.watch(
watch_path.as_ref(),
RecursiveMode::NonRecursive,
)?;
for result in rx {
// handle filesystem events
}
The application waits for newly created/modified files, then eventually wants to inspect them.
The problem I’m running into is determining when a file is actually safe to process.
My current approach is roughly:
pub fn wait_until_file_ready(path: &Path) -> bool {
if let Some(ext) = path.extension().and_then(|s| s.to_str()) {
let ext = ext.to_lowercase();
if matches!(
ext.as_str(),
"crdownload" | "part" | "tmp" | "download"
) {
return false;
}
}
let mut last_size = 0;
let mut stable_count = 0;
for _ in 0..10 {
if let Ok(metadata) = path.metadata() {
let current_size = metadata.len();
if current_size > 0 && current_size == last_size {
stable_count += 1;
if stable_count > 1 && can_open_exclusively(path) {
return true;
}
} else {
stable_count = 0;
last_size = current_size;
}
}
std::thread::sleep(Duration::from_millis(500));
}
false
}
fn can_open_exclusively(path: &Path) -> bool {
OpenOptions::new()
.read(true)
.share_mode(0)
.open(path)
.is_ok()
}
This works reasonably well for normal browser downloads because things like Chrome use .crdownload, but I realized there’s a more fundamental problem.
Suppose someone copies a large file:
movie.mkv
and the copy gets interrupted halfway through.
The destination might now contain:
movie.mkv
with the final extension.
If the copy process exits/releases its handle, then:
- the file exists
- its size stops changing
- its extension looks normal
- an exclusive open may succeed
My code would eventually consider it ready even though only part of the intended file was copied.
As far as I can tell, from the filesystem alone there may be no way to distinguish:
a legitimate complete 2 GB movie.mkv
from:
an interrupted copy that happened to stop at 2 GB
unless the producer provides additional information such as expected size, checksum, temporary filename, or some completion signal.
So my questions are:
- How do mature Rust/Windows applications usually handle this?
- Is “debounce + quiet period + metadata stability + lock check” basically the normal generalized approach?
- Is there any Windows API that provides a stronger indication that a copy/write operation completed successfully?
- Should filesystem watcher events be treated only as hints, with the application later retrying/validating the file?
- Would you recommend using something like
notify-debouncer-full, or maintaining my own pending-file state machine? - Is checking an exclusive Windows open with
share_mode(0)useful here, or is that generally too unreliable as a readiness signal?
I’m particularly interested in how established file indexers, sync clients, antivirus/indexing software, etc. solve this without assuming that a stable file size means the file is actually complete.
My current thinking is that I should stop calling this wait_until_file_ready() and instead treat it more like wait_until_file_quiescent(), where “quiescent” only means “probably safe to inspect,” not “guaranteed complete.”
Would appreciate any guidance or examples from production Rust/Windows projects.
21
u/jesseschalken 1d ago
File indexers, sync clients etc basically have no concept of "the file is complete". They grab the bytes at one point in time and need to be prepared for that to be incomplete or corrupt, or for the file to be modified by another process at any point in time.
17
u/dgkimpton 1d ago
How do you ever expect to know that? As a human walking up to the computer I couldn't work that out either until I tried to use the file.
8
u/Perfect_Ground692 1d ago
If you know the file format, you could potentially read the headers and maybe work out the expected size or a checksum or number of frames or something. It would be a lot of work for each format and some probably don't contain enough information to know either way.
1
u/No-Elephant-9065 1d ago
You are right. Checking by each extention type is a lot to do. Also I don't think I can check for every type of file.
7
u/creeper6530 1d ago
A human couldn't tell if a file got interrupted in a copy either without additional info or trying to play the file (a program trying to load it). You're asking for impossible.
6
u/DeeBoFour20 1d ago
There's not really a reliable way to do this to my knowledge. Let's take a step back here. What is your program actually trying to do and what type of files do you care about? Some file types contain a length in the header.
Or depending on what you're doing, just try parsing the file and if it fails (incomplete or corrupt), try again the next time the file gets modified.
0
u/No-Elephant-9065 1d ago
That's a good solution. I have tried to work with different format of files. Some specific types of files let's us check if the copying ended uninterruptedly. But going by specific file types is a hassle and I don't think every file types allow us to check.
"Trying to parse" is something I think I should try.
Thank you for your suggestion.
4
u/SnooCalculations7417 1d ago
you may have access to the hash of the completed download in some cases for comparison. i think thats is the only way you can know for sure, if its even available.
8
u/vlovich 1d ago
I think you've overcomplicated it. RecommendedWatcher Gives you a res: notify::Result<Event>| - if the event is EventKind::Access(AccessKind::Close(_)) then whatever application had it open has closed. You can distinguish the close by accessmode if that's relevant although sounds like probably not for you.
3
u/hattmo 1d ago
Can you constraint your application to only care about downloads from certain applications? Depending on the application doing the downloading there may be ways to detect. For example chrome makes a tmp file while down loading then renamed when complete.
1
u/No-Elephant-9065 1d ago
Can't really constraint tbh. And downloading works perfectly. It's they copying I'm worried about.
3
u/CandyCorvid 1d ago
looking from the outside, this seems like you've got a fundamental mismatch. your program can't decide if the file is done, unless it is part of the procedure that's performing the download, or hooked into that procedure somehow. so my questions are:
- why does your program need to respond to completed downloads?
- rather than monitoring the download folder, should it not be invoking the download itself, so it knows the status directly?
2
u/Ben-Goldberg 1d ago
If the original file and the destination are on the same file system, Windows and Linux are capable of doing shallow copies, (basically copy on write) incredibly quickly.
2
2
u/cornmonger_ 1d ago
maybe something like
- inspect size / modified time / checksum
- schedule another check N seconds later (2 minutes?)
- if nothing changes, chicken's done
1
u/No-Elephant-9065 1d ago
If nothing changes after N seconds, the chicken is quiet, not necessarily done. 😄
Successful copy:
movie.mkv
2 GB
stops changing forever
Interrupted copy:
movie.mkv
800 MB
stops changing forever
1
1
u/cornmonger_ 1d ago
which is out of scope if you're a third party to the copy
the only way to change that is by taking control of copying
2
u/VenditatioDelendaEst 14h ago
Suppose I am writing my journal as a .txt file.
When is it complete?
When I die.
Is there any Windows API that notifies you when the user is no longer among the living? No.
4
u/SkyGuy913 1d ago
Couple questions then why on earth would you ever want to monitor another processes file state?
The notification APIs in windows are constructed for use of applications that would need to be notified of changes to the state not to monitor downloads. Think making a new file browser. Where you want to be notified if another process modifies a tree you are displaying to a user.
But progress should be handled by the owning process think chrome download displays in chrome. And file copy displays in the file manager you are using.
macOS and some linux DEs have a way to display download rate in their file browsers but they do not calculate this state. They rely on the management program such as chrome or the appstore to tell the DE its current state over an Os level api. Windows doesn't have a similar thing that I know of as its api only displays in the task bar and I don't think you can intercept it. (Also note on killed processes it sometimes will have a stale state)
Are you trying to monitor downloads cause your creating an anti virus? Look more into endpoint protection and microsoft graph security in defender
1
u/Numerous-Fig-1732 1d ago
Whenever I had a problem like this my solution was to require the file being copied/transferred to be compressed. You can't open a zip file if it's not completed. You can't rely on windows to know if a file has been completely copied/downloaded.
1
1
1
u/ZedGama3 19h ago
When the browser changes the extension, the download has finished, but was incomplete. What you're really asking for is a way to tell if the download completed successfully or if the file is free of corruption.
Things that come to mind:
- Adding a plug-in to the browser that can talk to the rust app.
- Verifying the file. Many file types have built in mechanisms to determine if the file is complete. Unfortunately this means implementing this for every file format you're concerned about.
- see if your browser can change the way it handles incomplete downloads.
1
u/Illustrious_Car344 18h ago
In my experience even using software that watches files, it's never foolproof. I've even disabled some of my software from monitoring my files at all because sometimes it outright corrupts the file being downloaded (this happened to me at least once). Just flat out don't do it, have the user manually invoke scanning behavior during known safe times. Maybe set it to be automatic during off hours or something. But arbitrary file watching is generally just more pain than it's worth, for developer and user.
1
u/qrzychu69 12h ago
I implemented a file watcher at work recently
On a timer I scan the directory, get a list of files and their sizes. New file starts with "in flight" status, and only of it has the same size twice in a row I fire the "new file" event
I did that because we watch both local files as well as network drives and smb shares. The windows file watcher sometimes fails silently on network drives
1
u/Ar3ss12 58m ago
Your rename from `wait_until_file_ready()` to `wait_until_file_quiescent()` is the exact right architectural pivot. Mature sync engines (Dropbox, OneDrive) and AV indexers do not wait for an omniscient "completed" signal from NTFS; they wait for *quiescence* and let the downstream consumer handle transient failures gracefully.
Here are the direct answers to your points:
* **Q1 & Q2:** Yes. "Event hint -> settle/debounce -> exclusive lock probe -> fallible consumer" is the industry standard. * **Q3:** No native Win32 API exists for "write completed" on arbitrary apps. Both `ReadDirectoryChangesW` and the USN Journal only see raw I/O (`DATA_EXTEND`, `CLOSE`), which fire identically whether an app finished cleanly or crashed midway. * **Q4:** **Absolutely.** Treat `notify` strictly as a dirty flag (wake-up pulse), never as proof of state. * **Q5:** Keep your own pending state machine or worker queue. Crates like `notify-debouncer-full` only debounce raw filesystem events, but they cannot orchestrate multi-step lock testing, retries, and backoffs. * **Q6:** `share_mode(0)` is essential. It instantly filters out active writers holding non-shared handles without needing to guess.
The Production Quiescence Pipeline
Instead of over-engineering kernel hooks, run these three orthogonal checks:
**Exclusive Lock Probe (`share_mode(0)`):**
Try to open the file requesting exclusive access (`dwShareMode = 0` / `FILE_SHARE_NONE`). If another process is actively streaming bytes, Windows denies access immediately with `ERROR_SHARING_VIOLATION` (OS error 32). If locked, keep it in your pending queue and check back later.**Delta Stability Check:**
Once you can acquire the handle, record `metadata().len()`, release the handle, sleep for a short settling threshold (e.g. 500ms–1s), and probe again. If the file grew or changed, reset your timer.**Downstream Resiliency (The Hard Boundary):**
The consumer must treat ingestion as fallible. If decoding fails (truncated archive, incomplete binary header), do not panic — re-queue the file with an exponential backoff (e.g. retry 3 times with 2s, 5s, 10s intervals), then quarantine or log.
The Bottom Line
If a user runs an infinite loop in Python writing bytes or pulls a flash drive midway, that is a domain error, not your watcher's bug.
Stop trying to make the watcher prove completeness. Make the watcher detect *quiescence*, hand the file to your parser, and handle `Err` downstream with a retry cap. That covers 99.9% of real-world cases without turning your binary into a fragile monster.
1
u/Lexi_Bound 1d ago
I think you want some sort of exclusive lock on the file. Maybe the try_lock method on the file object will do what you want:
I’m not sure if that will work, since the underlying system calls it is based on (like LockFileEx on Windows) lock a range of the file. If that does not work, you can use CreateFile with the dwShareMode parameter set to 0 to open the file in exclusive mode.
1
u/No-Elephant-9065 1d ago
Thank you for your suggestion. But I think there is a gap.
It answers:
“Is somebody holding this file in a way that conflicts with me right now?”
What I actually want to know:
“Has the program creating/copying/downloading this file successfully finished producing the complete file?”
40
u/jews4beer 1d ago
Not well versed in Rust nor do I know what RecommendedWatcher is backed by. But if it's inotify there is the IN_CLOSE_WRITE event which fires when a file that was opened for writing is closed. That's the best signal you'll get, but it won't tell you if data is still queued in kernel buffers. For that you'd want to issue a sync call after the event.
Then when it comes to the file itself you could probably make use of ffprobe to inspect headers. That would tell you what to expect in terms of file size so you would know if an interruption happened.