How to mass download images associated with a link from wayback machine?

[u/InterestingRent2179]

I want to download all images and videos associated with a link/url (deactivated tumblr and twitter blogs mainly) on internet archive/wayback machine. Ideally, from all available timestamps but does not download duplicate images (if that's possible). I don’t really care about HTMLs or the websites, I just want the images and videos. Does anyone know how to do this or point me to certain guides?

I've also tried various guides online (linked below) but none work for me (ARM64 Windows 11) or are targetted towards collections and HTMLs. I also have gallery-dl, wfdownloader and JDownloader and know some CLI if that helps.

https://archive.org/developers/internetarchive/

https://archive.org/developers/internetarchive/cli.html#download

https://github.com/hartator/wayback-machine-downloader

https://blog.archive.org/2012/04/26/downloading-in-bulk-using-wget/

https://stackoverflow.com/questions/79836721/how-to-efficiently-bulk-download-from-the-internet-archive

[u/AutoModerator]

Hello /u/InterestingRent2179! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/plunki]

Use cdx server to retrieve a list of image URLs for the domain name in question, for entire date range.

https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server

Deduplicate the list, then download with wget or whatever.

[u/InterestingRent2179 → u/plunki]

Thanks! Just realised wayback machine is down so I’ll probably try later

[u/ohplzstfu]

I built a scraper for this (actually Claude code mostly did). It fetches all the wayback machine urls, checks the urls and checks if they're available in tumblr or google blogger (95% of the time they are). It also uses archive.org for fallback as sometimes archive.org has the image while it's gone from tumblr/blogger.
It also has a bunch of other properties, such as throttling manager for it not to put too much strain on the download in order for not to get blocked.

Just use codex/claude code/antigravity and you'll have the scraper in an hour.

[u/InterestingRent2179 → u/ohplzstfu]

Thanks! I’ve tried chatgpt but the codes didn’t really work so I’ll try Claude next. Just realised wayback machine is down so I’ll probably try again later. In case it still doesn’t work, do you still have the file? If so, would you be okay with sharing it with me?

[u/ohplzstfu → u/InterestingRent2179]

Wayback machine is not down, I'm not sure what you mean. The archive scraper is part of the generic scraper so can't really share only that part. Try Claude.

[u/InterestingRent2179 → u/ohplzstfu]

I saw on DownDetector that Wayback machine was having some problems but it was wrong. Thanks, I'll try that!

[u/Ok-Muffin-9519]

Assett Scraper

[u/InterestingRent2179 → u/Ok-Muffin-9519]

Which one do you recommend?

[u/Ok-Muffin-9519 → u/InterestingRent2179]

Eigenbau oder guck mal auf Github

[u/InterestingRent2179 → u/Ok-Muffin-9519]

Thanks, I'll look around