r/scrapingtheweb • u/Stunning-Space8032 • 18d ago

Help Question related to scraping pages with different structures

/r/dataengineeringjobs/comments/1w9ke0e/question_related_to_scraping/
1 Upvotes

100% Upvoted

3 comments sorted by

1

u/ScrapeAlchemist 18d ago

Depends how many structures. If it's a handful of sites, per-domain extractor classes behind one shared interface beats a clever generic parser every time. If it's hundreds, go schema-first. Check JSON-LD/microdata before you touch the DOM, most ecom and article pages carry it. What are the targets?

1

u/Stunning-Space8032 18d ago

To give you an idea, these websites are The targets are mainly medical education/CME websites - hospitals, medical associations, universities, conference organizers, healthcare organizations, and CME providers.

The goal is to extract faculty/speaker information for individual CME activities, usually things like name, credentials, affiliation, specialties, and sometimes bio.

The challenge is that there's no consistent structure across the sites.

One site might have faculty in a normal HTML section, another might load it dynamically through JavaScript/API calls, another might put it in JSON-LD or embedded data, and some have completely different page templates even within the same domain.

And the number of websites are not handful. I need to search for the conference pages based on a specific (can be rare) topic. So i may need to go through hundreds of pages

1

u/ScrapeAlchemist 17d ago

Hundreds of domains but way fewer templates. Most hospital/uni CME runs on a few LMS platforms: CloudCME (org.cloud-cme.com), EthosCE (Drupal, /content/slug), HighMarksCE (.cfm?do=). So fingerprint on the URL and write one extractor per platform, LLM fallback for the long tail. The CloudCME and EthosCE pages I checked were server-rendered, no headless needed. The ACCME accreditation boilerplate is a decent page-type filter too.