FBScrapeIdeas

Overview

FBScrapeIdeas grew out of a simple research workflow - reading public discussions on a given topic, summarizing common pain points, and surfacing ideas worth building. Doing this by hand is slow and inconsistent. I wanted a tool that could scrape Facebook group discussions, run them through an AI analysis layer, and produce a structured output I could actually act on. It ended up being one of my more starred Python projects.

How the pipeline works

The tool uses Playwright for the browser automation layer because Facebook's frontend is hostile to traditional scrapers and Playwright handles modern web app patterns more reliably than alternatives. Self-healing selectors keep the script working when the DOM shifts between releases. Once data is collected, Gemini AI analyzes the discussions for sentiment, recurring themes, and product opportunities. Everything runs locally - the analysis can stay offline if you swap in a different model.

Why it stayed CLI-first

I considered building a UI but kept resisting the urge. CLIs are honest about what they do, easier to automate, and more useful in the kind of research workflows this tool was built for. The audience that finds this project useful is not afraid of a terminal. Adding a UI would have made it look more polished without making it actually more useful.

What I learned

This project taught me that browser automation is mostly about anticipating change. Facebook's interface shifts often enough that a brittle scraper would have died within weeks. Building self-healing selectors, sensible retry logic, and clear failure modes made the tool sustainable. The AI layer was the easy part. The discipline of writing automation that survives without supervision was where the real engineering went.