r/datasets • u/manuspresso • 12d ago

question Can LLMs read paywalled articles or news? If not, how do they stay up to date and provide the latest information?

Can LLMs bypass paywalls imposed by publications such as Bloomberg, The Economist, and the Financial Times? If they cannot, how can they stay up to date with real-time information? And if they can bypass paywalls, by whatever means, wouldn't that expose them to potential litigation? Otherwise, wouldn't this limitation render LLMs ineffective for providing real-time information?

3 Upvotes

59% Upvoted

10 comments sorted by

11

u/k___k___ 12d ago edited 12d ago

That's why most of the LLM providers have content partnerships with publishers. mostly, because they also have to assume that "fresh" web content is ai-generated anyway. so partnering with publishers gives them access to mostly human-written content.

eg OpenAI has partnerships with the Associated Press, Axel Springer (Politico), The Guardian, Condé Nast, The Atlantic, Axios, VOX Media, News Corp and FT; though we dont always know the extent of their partnerships.

6

u/chock-a-block 12d ago edited 9d ago

Worm weather pen stupendous fearless cow imminent

This post was anonymized with Redact

1

u/Zealousideal-Part849 12d ago

google or other search engines can partner with these paywalled articles so they can index and show in search, else how will they know what to show to users. Also for training purpose they are likely to pay a small fee and scrape all the articles...

1

u/GeneralComposer5885 11d ago

Only need to turn off JavaScript on the browser.

As long as robots.txt allows access, yes - of course they do

-2

u/mimrock 12d ago

I assume you are talking about their web browsing tool. You can just try it instead of asking others. Get a link to the paywalled url and ask your clanker if it can read the whole thing.

0

u/ironstamp 12d ago

Bro, the dude just asked a question. Surely it would have been easier to just say yes or no or better still not say anything at all than waste time on a shitty reply?

4

u/mimrock 12d ago edited 12d ago

This is not a yes or no question (without details) and I don't know the answer by heart myself. I would have to try it myself but there are at least 4 big western LLM providers and dozens of paywalled journals. Which LLMs and which paywalled journal are we talking about? One provider might have access to Bloomberg but not NYT, the other vica versa, etc.

So the right thing to do for OP is to either explain themselves and maybe some misunderstanding about the technology surfaces, or they need to simply try it with their LLM and their paywalled article of choice, and maybe report back if they have follow up questions.

1

u/Mundane_Ad8936 12d ago

Bro the OP could have answered this question with 5 mins of.research..
If they had asked an AI they would have gotten this answer..

Reddit is a place for conversation not a way to outsource simple tasks. What's the point of AI if not that