fix: NorthHertfordshireDistrictCouncil - migrate to Netcall Liberty Create portal#1975
fix: NorthHertfordshireDistrictCouncil - migrate to Netcall Liberty Create portal#1975InertiaUK wants to merge 1 commit into
Conversation
…reate portal
NHDC has moved its bin-collection lookup from
apps.cloud9technologies.com/northherts/citizenmobile/mobileapi/{UPRN}/
(now returns 404) to
waste.nc.north-herts.gov.uk/w/webpage/find-bin-collection-day-input-address
which is a Netcall Liberty Create portal.
There is no public UPRN shortcut on the new portal - submit URLs carry
an auth signature bound to the session-issued webpage_token (direct
GET with id=<record_id> returns 403). The form is also entirely
client-rendered (the initial HTML contains no <form>), so a requests-
only implementation would have to replay 30+ presenter params and is
extremely fragile. Selenium is the pragmatic path and the old fixture
already declared web_driver: http://selenium:4444 anyway.
New scraper flow:
1. driver.get the input form page.
2. Type postcode into input.relation_path_type_ahead_search.
3. Wait for div.relation_path_type_ahead_results_holder ul.result_list
li[data-id] to render.
4. Pick the li whose text contains the paon (case-insensitive
substring match; falls back to the first result if the caller
doesn't provide a house-number match).
5. Click input[type=submit][aria-label='Select address and continue'].
6. Wait for the show-details result URL, then parse bin-type headings
(Cardboard & Paper / Food Waste / Non-Recyclable Waste / Mixed
Recycling / Garden Waste / Refuse / Glass) paired with the
following 'Next collection' date. Strip ordinal suffixes and
parse '%A %d %B %Y'.
Fixture updated:
house_number='Stewards Flat', postcode='SG5 1PZ'.
Verified live: returns 4 bins (Cardboard & Paper 23/04/2026,
Food Waste 23/04/2026, Non-Recyclable Waste 30/04/2026,
Mixed Recycling 08/05/2026).
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 52 minutes and 38 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Included in May 2026 Release PR #1992. Closing. |
Problem
The old scraper targeted
https://apps.cloud9technologies.com/northherts/citizenmobile/mobileapi/{UPRN}/which now returns HTTP 404. NHDC has moved its bin-collection lookup to a Netcall Liberty Create portal at:There is no public UPRN shortcut on the new portal — submit URLs carry an
authsignature bound to the session-issuedwebpage_token(direct GET with?id=<record_id>returns 403 Forbidden). The form is also entirely client-rendered (initial HTML contains no<form>), so a pure-requests implementation would have to replay ~30 Liberty presenter params per call and is extremely fragile. Selenium is the pragmatic path — the old fixture already declaredweb_driver: http://selenium:4444anyway.Fix
Selenium-driven typeahead flow:
driver.getthe input form page.input.relation_path_type_ahead_search.div.relation_path_type_ahead_results_holder ul.result_list li[data-id]to render.<li>whose text contains thepaon(case-insensitive substring match; falls back to the first result if the caller doesn't provide a house-number match).input[type=submit][aria-label='Select address and continue'].show-detailsresult URL, then parse bin-type headings (Cardboard & Paper/Food Waste/Non-Recyclable Waste/Mixed Recycling/Garden Waste/Refuse/Glass) paired with the followingNext collectiondate. Strip ordinal suffixes and parse%A %d %B %Y.Fixture change
Required — the old fixture was UPRN-only and the new portal doesn't accept UPRN input. Updated to:
Verification
Live run against the fixture returns 4 bins: