Playwright

Playwright drives a real browser from code. Open a page, click, type, wait for something to appear, take a screenshot, read what the page requested over the network. It comes from Microsoft, it is free, and it works in Node, Python and a few other languages.

How I use it

Three jobs. The first is screenshots: a script opens a list of landing pages at phone and desktop widths and saves an image of each, so I can compare twenty pages in a folder instead of twenty tabs. The second is testing my own sites before a deploy: open every route, check for the strings that must be there, fail if one is missing. The third is the one people overlook. Playwright records every network request a page makes, so when a web app has no public API, I let the browser log in and click around, read the calls it makes, and then write a script that makes those calls directly. That has turned more than one tedious web interface into a few lines of Python.

What surprised me is how much less flaky it is than the tools before it. Waiting is built in, so a click on an element that has not rendered yet waits instead of failing. Codegen, which records your clicks into a script, is a good first draft generator. What annoys me is the browser binaries: they are large, they must match the library version, and a machine where the two drift apart fails with an unhelpful error. Some sites with bot detection also refuse headless browsers, and the workarounds are fragile. The Python bindings get new features a little later than the Node ones.

Who it is for

Any developer who needs a browser in a script. If you write end-to-end tests, this is the current best choice. If you only need to fetch HTML, a plain HTTP client is lighter and you should use that instead. Skip it if your target site actively fights automation; you will spend your time on evasion rather than the task.