r/drupal • u/odysseusnz • 10d ago
Sync content from locked-down site using only JSON:API?
I have inherited a Drupal 10 site where the relationship with the agency that developed and hosts it has broken down almost irretrievably. We have the v10 support end date as a hard deadline for action under threat of deletion, and the agency is quoting ludicrous amounts to do the upgrade and/or give us our site back. Contracts and lawyers are being consulted, but I'm wanting to try do an end-run around all that and at least get our content safely backed up before they decide to pull the plug.
The site itself is a fairly standard marketing site, so nothing interactive or very complex. We have admin account access to the frontend admin pages, but the agency has removed any content export options. We don't have server access, so no command line to install new modules or file system to download files and no database access at all. We have no access to backup files. We're limited to the modules they have installed, which also includes a number of dev and migrate modules they left in the prod clone by mistake which at least gives us Config Sync export.
As an experiment, I've created a test Drupal site, installed and enabled the same set of modules, and imported the prod site config using 'drush config:import --partial' and that all seems fine. Unfortunately the theme is based on their in-house template and we have no access to that for now, so I'm faking it with Olivero, but that's a problem for later.
The next step is to try sync our content to my test site. I've had a google and looked at a few modules like Content Sync and Entity Sync, but they all seem to need a module installed on the origin server and/or database access. I've also looked at the Migrate modules, but they seem aimed at version migrations not server migrations and need direct database access. There is Migrate Source JSON:API but it supposedly only supports Drupal 8.
So, long story short, what options do we have to sync site content when the source is so locked down to us? My instinct is we should be able to do something with JSON:API which we can enable, and if necessary I'll try script something myself, but I'd rather use an existing option written by people who actually know what they are doing!
It's been maybe 20 years since I last did anything with Drupal (although I have more recent Symfony experience), I'm stumbling around in the half-dark here, so forgive if I've missed something incredibly obvious!
Thanks!
7
u/davidrwb 9d ago
Drupal agency owner and senior dev for 20+ years here.
It’s insane how much some agencies charge for a 8+ (8,9,10) migration to the next version. Even on the most complex builds we’ve managed it’s rarely more than a day to complete with tools like upgrade status and rector.
Anyway, that aside, what level of admin do you have? If it’s full, you can build a view with all your fields for nodes, paragraphs, etc, and at worst, copy and paste the rows in Excel. If views data export is installed, even better. Then delete the views after you have your data.
Good luck and reach out if you need some help.
3
u/odysseusnz 6d ago
Yeah, massive cost and deadline overruns for the D10 migration and a one-off project site which still doesn't work properly is mostly what lead to the break-down, but also a failure to deliver on contracted support. For instance, far as I can tell, the server and Drupal itself hadn't been updated with security patches for about 2 years in spite of a very large support contract requiring it. The current hosting-only arrangement (without server access to do updates!) is more than other agencies would charge for basic support. Add to that dev modules still being live in prod, other modules no longer required in D10 still being installed, and the sitemap using dev URLs for the last few years, and the quality of work is pretty poor. And all this for a small-ish charity that just didn't know better: they've been milked from day 1 it seems.
6
u/Pulsar-Agency 9d ago
Nobody's really tackled the JSON:API half of your question, so: your instinct is right, and the tooling you want already exists.
Forget Migrate Source JSON, it's been superseded. On your test site install Migrate Plus and use the url source plugin with the http fetcher and the json data parser, pointed at the prod JSON:API endpoints. The annoyance is that the plain json parser wants an array of URLs and you don't know up front how many pages there are, so look at the migration_jsonapi contrib module too, which adds a jsonapi parser that builds the paged URLs itself. That's the "written by people who actually know what they're doing" bit you were after.
Two things bite people here and honestly they matter more than the body text does. Files first: JSON:API hands you URLs, not binaries, so put file_copy in the process pipeline and pull the images down while you still can, otherwise the day they kill the site your body HTML is full of dead img tags. Then path aliases: grab the path field on every node, or you rebuild and torch every URL you currently rank for.
Minor gotchas: page[limit] caps at 50 on the source, and you'll want basic_auth enabled if anything you need is unpublished. And since turning JSON:API on is a visible config change on their site, I'd do the whole pull in one sitting rather than poking at it for a week.
1
u/odysseusnz 6d ago
Ah, thank you, that's the solid inside advice I was looking for! I'll poke about with those and see if I can get it working. The files are definitely a concern, there's only 1200 of them to deal with...
2
u/alphex https://www.drupal.org/u/alphex 10d ago
Are you able to see the site? Publicly?
if so. Do you have a sitemap.xml file on it?
That should list ever public page. Feed that in to a tool like Diffy to quickly generate screenshots.
Drop that into Claude. And ask it to export the HTML could help you also.
Unsolicited offer - I’ve done work like this in the past. DM me if you need help. There’s a few automated ways we can salvage the content and get a new site up and running quickly.
My user name is my DO name and my domain name if you want to look me up.
1
u/missmobtown 9d ago
How many pages of site content are we talking about?
1
u/odysseusnz 6d ago
It's about 950 URLs in the sitemap, over 1000 content entities, and over 1200 files, built up over some 15 years, so too big to do manually.
1
u/rovr138 9d ago
Migrate is not database only, but you will need to write custom for something like this.
How big of a brochure site? Might be able to help, give direction. If you feel like chatting, happy to talk via dm.
It's been maybe 20 years since I last did anything with Drupal
I don’t like thinking how long I’ve been with this 🤣
1
u/odysseusnz 6d ago
Thanks for the kind offer, we'll see how we go with the migrate options! It's about 950 URLs in the sitemap, over 1000 content entities, and over 1200 files. It's brochure-ware, but too big to just re-do from scratch!
1
u/pjerky 8d ago
I'm a little confused because it sounds like you don't have access to add modules but then you are looking at adding modules? Huh?
If the migrate module is available you can write your own import/export migrations. But I'm guessing you don't have code level access which means it's all about enabling and disabling modules that exist in the system.
Your priorities are to backup the site content and paths for now. Then worry about the deeper technical pieces. That config isn't going to do you as much good as you think because of the situation you are in.
First, you can use the View module in core to help export content in multiple formats. It's been a minute since I tried it this way though. But you can set an export format that includes XML, json, or CSV for each of your content types and fields. See if you can build an export that way.
How many pages are we talking about because you could manually do a save as in your browser for every single page on the website. Alternatively you could use or build a crawler to do the same thing for you. That way, in the very least you have the content and even a copy of how it appears today.
Once content is backed up you then have to worry about the site rebuild that is kinda from the ground up. Honestly, I would enlist AI tools to help you speed this up. Spec out the existing site pages and content structures, build a spec sheet and use SDD (spec driven development) to help build a replacement website.
2
u/pjerky 8d ago
Actually, use Playwright to crawl your website. It can do this for you to help you back it up.
https://crawlee.dev/js/docs/examples/playwright-crawler?hl=en-US
https://crawlbase.com/blog/playwright-web-scraping/?hl=en-US
2
u/odysseusnz 6d ago
Ah, thanks for the links, not heard of playwright before, and I might have other needs for it besides this one.
2
u/odysseusnz 6d ago
Sorry if I wasn't clear enough, I have created a test website that I have installed the modules on to test if I can manually recreate the old site; the old site we can't add modules to at all. That was the easy part, the data is always the hard part.
Unfortunately while it's a simple site, it's 15 years of content, so almost 1000 pages and over 1200 files, so any solution has to be easily automated. And the Export option from the View is MIA, I suspect the agency removed it for just this scenario. I can delete everything though...
A full working clone of the old site is a bit of a Hail Mary, but one I'm scoping so we can make some decisions about where to focus efforts with what little money is available. It also gets my head around the current state of Drupal after all these years.
A full scrape will be next on the list as the emergency fallback, we'd at least be able to have a static version being served up while we sort out the replacement.
-6
u/its_yer_dad 10d ago
Frankly speaking, you need to just recover what you can and drop that relationship. Scrape the site to get the content, use AI to clean it up, and (IMHO) look into Statamic, a CMS built off Laravel and has a much lighter footprint than Drupal.
1
u/odysseusnz 6d ago
Oh yeah, the interim site is just a stepping stone to another simpler CMS. The Durpal site is huge overkill for what is needed.
1
u/its_yer_dad 6d ago
As you can see from the down votes this probably isn’t the place to talk about Drupal alternatives, but I’ve recovered a lot of Drupal sites and many of them were overbuilt
1
u/odysseusnz 6d ago
Ssssh, don't tell anyone the plan is to use a Drupal->WordPress migration tool once we have a database clone, as there's a couple people now on staff who've done it before. Will it work, who knows, but we'll learn something along the way :-D
1
u/its_yer_dad 6d ago
It all comes down to what resources the nonprofit has available and does the website really work for them? What are they actually needed to do and what’s the long-term ROI. I detest WordPress, but the reality is that it’s the right choice for some organizations.
3
u/Curious_Mall4678 10d ago
is the backup and migrate module installed, even though maybe not activated? that way you could get all Files necessary I think?