Client: [Client Name]
Prepared by: QiAlly Document Services
Consultant: Cody Rice-Velasquez
Date: November 2025
1. Project Goal
Extract approximately 8,000 member listings (≈ 571 pages) from the MIBOR Member Directory, producing durable artifacts (HTML/text/screenshots) per page for later parsing and structured analysis.
Deliver a clean, automatable process that preserves login sessions, mitigates timeouts, and yields exportable CSV/JSON datasets for business intelligence and CRM ingestion.
2. Work Performed (Chronological Summary)
Step 1 — Site Access & Analysis
- Logged into
member.mibor.comand confirmed listings appear in a card-based directory with server-side pagination. - Verified directory uses DotNetNuke (DNN) + Telerik RadListView, enforcing 20 results per page.
Step 2 — Initial Manual Capture
- Captured Pages 1–50 with full-page screenshots.
- Confirmed session resets occur beyond Page 30 and automation instability begins after Page 50.
Step 3 — Deliverables Clarified
- Output formats selected: HTML/Text or Screenshots per page.
- Progress logged in
mibor_progress_up_to_page50.md.
Step 4 — Technical Limitations
- No direct page-jump feature; session resets after prolonged pagination.
- Browser automation crashes intermittently at high page counts due to DNN’s server-side token refresh.
- Manual capture remains the fallback plan for failed pages.
3. Automation & Engineering Plan
- Developed and tested a custom browser-automation script with pause/resume, cookie persistence, and randomized human-like delays.
- Validated stability through 50-page runs; average ≈ 16 seconds per page.
- Identified intermittent crash after extended pagination due to asynchronous session refresh failures.
- Next phase: debug automation to improve fault recovery; if instability persists, complete remaining pages manually with cost adjustment.
Outputs
| File | Description |
|---|---|
scrape_mibor_cdp.js | Final automation script (Playwright-based) |
mibor_members_full.csv/json | Structured dataset |
mibor_pages/ | Saved HTML/screenshot pages |
checkpoint.json | Resume state for restart after failure |
.pw-profile/ | Persistent login session |
4. Research & Testing (Non-Billable)
| Category | Description | Hours | Billable |
|---|---|---|---|
| Bot / Agent Testing | Playwright & Notion integration testing | 1.5 h | ❌ |
| Framework Comparison | Selenium vs Playwright vs Puppeteer | 1 h | ❌ |
| Session Experiments | Cookie persistence & CDP connections | 0.5 h | ❌ |
| Pagination Research | Timeout and session-life testing | 0.5 h | ❌ |
Total Non-Billable Research: 3.5 hours
5. Billable Work & Pricing (Revised)
| Description | Rate | Hours / Basis | Amount |
|---|---|---|---|
| Initial Research & Planning Fee (Setup + Testing) | flat | — | $50 USD |
| Automation Build, Debugging & Verification | $40 /hr | 6 h | $240 USD |
| Manual Capture Contingency (if automation fails for remaining pages) | $25 /hr | up to 10 h | $250 USD max |
| Packaging & Deliverables (CSV + JSON exports) | flat | — | $50 USD |
| Estimated Project Total (Hybrid Range) | 590 USD |
Payment Terms:
- $50 Research Fee due upon acceptance of proposal.
- Remaining balance due upon final delivery of data package, based on final automation vs. manual ratio.
6. Technical Notes
- Estimated 14 records per page × 571 pages ≈ 8K records total.
- Automation stable through Page 50; debugging in progress for extended runs.
- Manual fallback procedure ensures full dataset delivery even if automation cannot complete the run.
- Lawful archival use only — no redistribution of directory data.
7. Next Steps
- Client confirms preferred output format (PNG screenshots or HTML/Text).
- QiAlly continues debugging automation to stabilize beyond Page 50.
- If unresolved, fallback to manual capture for remaining pages (at adjusted rate).
- Export final dataset (CSV/JSON) and provide deliverables via secure download.
- Optional: set up Cloud-Automated Run for future unattended refreshes.
8. Acceptance
| Client Signature | ___________________________ |
| Date | ___________________________ |
9. Summary Reflection
QiAlly engineered a resilient hybrid workflow balancing automation speed with manual reliability. Even in the event of system instability, the process guarantees delivery of the full MIBOR directory dataset, ready for analysis or CRM import.
This engagement underscores QiAlly’s commitment to accuracy, transparency, and completion — adapting technology and human oversight to ensure dependable results.
Prepared by QiAlly Document Services — Professional Data Preparation & Automation Solutions