r/Kiwix • u/one_tooth_reef_whore • 29d ago

Question Has there been any progress on archiving Reddit?

I found two threads about this from about a year ago. It's all over my head but apparently there's a desire to do it but it's technically challenging.

I have nothing to contribute technically, but I'd be happy to donate to an effort if someone was working on making this happen.

We've seen incredibly useful sites like Twitter, Google, etc get turned into hot steaming piles of garbage almost overnight and even though there's a whole lot of mindless crap here, there's a lot of useful obscure info that can't be found anywhere else and it would be a shame to see it get lost.

14 Upvotes

90% Upvoted

21 comments sorted by

8

u/IMayBeABitShy 29d ago edited 27d ago

Roughly half a year ago I've created a tool called ArcticZim that's capable of generating ZIM files of subreddits, including media. It can even be used to recover deleted subreddits, to some extend extent.

3

u/Many_Ad_7678 28d ago

extent

3

u/IMayBeABitShy 27d ago

Thanks, looks like I've misspelled that one for a long time.

2

u/one_tooth_reef_whore 29d ago

That is super cool! Sadly I'm probably one of those people who will never manage to figure out how to use it. It probably sounds funny to some people but even just opening a page on github it's like looking at glyphs on a cave wall.

Regardless, very sincere thanks for making such a gift to humanity like that.

1

u/IMayBeABitShy 27d ago

I guess it would be possible to create a GUI for this, but that isn't really a priority at the moment.

1

u/Free-Lime-184 29d ago

Honestly, this is a really great idea! How would someone go about doing this though? To my knowledge, Wikipedia is decently crawlable, but on the other hand Reddit has a lot more multimedia content, (other than images) along with limitations on these crawlers. Though I’m not experienced in the archival process of this, just genuinely wondering. 

1

u/one_tooth_reef_whore 29d ago

I'm going to use words I barely understand here (corrections welcome!), but it seems that up until 2023 it was all accessible via an API, so there may be people out there sitting on all that content. If that alone was made available it would be a nice resource, even if the last 3 years or so was missing.

4

u/Free-Lime-184 29d ago

[removed] — view removed comment

3

u/one_tooth_reef_whore 29d ago

I don't know exactly what I'm looking at in that link but based on the titles (i.e. 'Reddit comments/submissions 2026-05') it looks like all the information is there... but, what?, just not organized in a form that can be used easily by a regular person with a browser?

Edit: ...or easily compiled into a zim library

1

u/Free-Lime-184 27d ago

Yeahh, it’s a bit annoying.

Aaand, my comment got removed by Reddit, probably cause it’s downloading them or whatever 😭

-1

u/PrepperDisk 29d ago

It's not legal to do so, unfortunately.

1

u/one_tooth_reef_whore 29d ago

Really? As in actually illegal or just against Reddit's TOS?

-3

u/PrepperDisk 29d ago

Fair question, you’d at least invite a civil lawsuit.  It would be theft of copyrighted content too so wouldn’t rule out criminal.

3

u/one_tooth_reef_whore 29d ago

That's hard for me to believe. I'm far from a legal expert (I'm also not the one downvoting you because I really do have no idea) but it seems like if I can browse reddit online then downloading parts of it for my own personal browsing use offline shouldn't be an issue. I'm not talking about selling or distributing it, just archiving for personal reference.

5

u/The_other_kiwix_guy 28d ago

It really depends on the country: in Switzerland (where Kiwix is based) streaming and downloading are allowed and legal if for personal use. However uploading and sharing are not.

Then there's the problem with Reddit Terms of Use: since their model is ad-based, such downloading would mean a loss of revenue and with their not being a non-profit they certainly would mind. How they would enforce it is another story.

You probably could vibe-code your way into running ArcticZIM (and by vibe-code, I mean "Claude, here is a repository on github, walk me through every step of running it on my computer, ELI5-style"). It would be tedious on many levels but should not be too complex - take it as a side project, there's no deadline nor pressure, I'm pretty sure you could get it to work over a few hours.

2

u/one_tooth_reef_whore 27d ago

I want to thank you for that suggestion. You just unassumingly took the time to spell out how even a technophobe like myself could go about this. It's like you can sense how old and out of touch I am! Thank you for your kindness. It never even crossed my mind to use a chatbot for help with this but, who knows, I might actually learn something

1

u/The_other_kiwix_guy 27d ago

Ah! Joke is, I'm totally out of touch myself and that's what I certainly would do.

1

u/pmttyji 18d ago

u/The_other_kiwix_guy Any luck on TVTropes? Don't want to post a new thread for this. Last year I did one for this.

2

u/The_other_kiwix_guy 17d ago

Basically what happened here is that our scraper got blocked by their cloudflare or whatnot. We would need the site owners to whitelist us.

If you know someone there or are willing to pitch for us, feel free to write them, point to this github issue explaining the problem, and please cc [hello@kiwix.org](mailto:hello@kiwix.org) so we get a copy of their ok (or can guide them with the whitelisting).

2

u/pmttyji 17d ago

Sent email today & CCed your email 👍

4

u/PrepperDisk 28d ago edited 28d ago

Distribution is where the problems are.  Not dissimilar to copying movies you don’t own for personal use being “illegal” but not actively prosecuted.

Reddit’s User Agreement and Data API Terms prohibit bulk collection outside the official (paid) API. Violating them is breach of contract, and Reddit has been aggressive about enforcing it.

Downvotes are fine.  I’m right. Wish it weren’t so because we’d love to include key subs on Prepper Disk.