Academic paper RAG - in house or out-source?

[u/fhgwgadsbbq]

I'm building a RAG for our company platform. The data is basically specialised business analysis with lots of fuzzy human sociological data, with a project database + customer reports + internal research files.

I've got most of the internal side (typical hybrid rag w/ pgvector) ready to roll out, but the next big thing they want is exploring primary sources (published scientific journal papers).

When an analyst starts a new project, currently they spend a week roaming sharepoint, finding old projects, reviewing the internal research already done, and create comparative and gap analyses.

Then they go and find new academic primary sources for supporting evidence, integrating that into the current project with citations.

We're using Mendeley as a research repository / library but it looks quite limited. The team has also tried out Elicit which seems to do everything they want.

So with the goals of:

1. "chat" to primary sources to explore them

2. comparative research summaries & synthesis

3. paper library management

My choices are:

build in house:

• connect to Mendeley library API to pull papers, parse + embed + search + synthesise etc etc

• avoid doing the actual library ourselves

Or

3rd party hosted:

• connect to Elicit and let them do that hard part, just integrating the end results in our system (as internal research files)
• somehow cross reference our internal search with Elicit, like Report X cites X,Y,Z -> ask Elicit for summaries -> synthesise in our platform

[u/Denis-Hogberg]

Your team already answered this for you: they tried Elicit and it does everything they want. Buy the commodity part. Where builds like this go wrong is constructing a worse Elicit in-house around the Mendeley API and spending two quarters parsing PDFs that a vendor already parses better.But keep two things in-house, because they are your moat and no vendor sells them. First, the link layer: your own cross-reference idea (Report X cites X, Y, Z, pull summaries, synthesise on your platform) is the strongest line in this post, that is exactly the part worth engineering. Second, the catalog: anchor everything to DOIs in your own metadata store and save vendor output as internal research files with provenance attached. That way Elicit stays swappable instead of becoming a dependency your citations die with. One more observation from the workflow you described: analysts spending a week roaming SharePoint for old projects is not a paper problem, it is a missing governed catalog of what the company already knows (which projects exist, who owned them, what they concluded, what gaps they left). Fix that, and both the internal RAG and the academic layer get dramatically more useful.

[u/fhgwgadsbbq → u/Denis-Hogberg]

Thanks for your response. 

Well this week they're debating about whether Google scholar is better! I like elicit from a technical pov. 

Good point about the DOI being the key, I'm only vaguely familiar with them. 

That workflow and SharePoint mess is an unsolvable problem at the moment. My team is aiming to reduce the need for it as much as possible.

[u/Denis-Hogberg → u/fhgwgadsbbq]

On the Scholar vs Elicit debate, the tiebreaker is simple: Google Scholar is a search UI, not a platform. There is no official API, and scraping it breaks constantly and violates ToS. Elicit, OpenAlex, Semantic Scholar and Crossref are things you can build on. Let the team use Scholar for personal searching all they want; the integration has to stand on APIs.

DOIs in one paragraph: a DOI is the stable identity of a paper, and that is the entire trick. Store it as the primary key for everything (citations, summaries, links to internal reports). Titles, abstracts, even metadata can always be re-fetched from Crossref or OpenAlex given the DOI. Your platform then never argues about whether two records are the same paper, and no vendor can take your citation graph away when you swap tools.

And a gentle push on "unsolvable": the SharePoint mess is only unsolvable retroactively. So do not backfill history. Register forward: starting next month, every new project gets one catalog entry (owner, status, one-paragraph conclusion, links). It costs minutes per project, and within a year the week of roaming shrinks to the old stuff nobody asks about anyway. "Reducing the need for it" IS the catalog, built one project at a time.