firehose/app/priv/blog/engineering/2026/07-30-disposable-end-to-end-tests.md

%{ title: "Disposable End to End tests", author: "Willem van den Ende", tags: ~w(), description: "", published: false }

I quite like playwright for the simplicity of writing End to End tests through the browser. Last year I realised that a coding agent (Claude Code, but Pi and local models can handle browsers as well) makes it cheap to write tests. Initially I didn't like the tests produced, but re-writing one by hand, maybe with some assistance, deleting ones that were superflous and then having the others rewritten in the style of the golden example was pretty painless.

Keeping them around does come at a cost, though.

1. Don't let people do what computers can do (unless it is the better way to learn for the people)

2. Don't let an LLM do what a simpler machine learning system or symbolic AI can do.ssh://git@gitea.apps.sustainabledelivery.com:3022/QWAN/dirtytests_cs.git

3. Don't let any AI do what deterministic code does better.

4. Make work fun and engaging for people

5. Tends to be faster and more energy efficient than 2. With people you never know, they can be very efficient and circumvent problems instead of attacking them head on.

In the end Pi could not access the browser through my sandbox, I have some configuration updates to make. But my 27B model used curl, found how to use the CSRF token in the page and made it through.

Handover message from the tool

All tests pass, clean compile. The script is ready at scripts/deploy-and-test-admin.sh.

Usage:

  ./scripts/deploy-and-test-admin.sh --password 'MyP@ss123!'                                                    

What it does:

1. ssh dokku — sets ADMIN_EMAIL / ADMIN_PASSWORD on the Dokku app
2. git push dokku — deploys (migrations run, admin is provisioned)
3. Waits for the app to be ready (polls /users/log_in)
4. Logs in as admin via curl, verifies redirect to dashboard
5. Checks /admin/tenants is accessible as super admin

Options:

• --password — admin password (prompts if omitted)
• --app-url — override the app URL (auto-resolved from git remote)
• --app — override Dokku app name (default: wereview2)
• --skip-deploy — skip the git push, just test login

Final test run

The first test run failed, my coding agent had tried to do something clever, reviewing the script: derive the url from the git remote of the repository. 27B clearly has been trained on dokku, and remembered it well. That does not work yet, as I had not set up the port mapping. Since I have time on desk now, I decided to do the riskiest part first, the rest is following my notes on previous deployments (yes, I remain, as ever, optimistic when deploying new applications).

export APP_URL="http://wereview2.apps.sustainabledelivery.com:5000/"
willem@sequential:~/dev/spikes/elixir/wereview-2$ ./scripts/deploy-and-test-admin.sh 

=== Super Admin Deploy & Test ===
  Email:  <email> 
  URL:    http://someurl.apps.sustainabledelivery.com:5000/
  Dokku:  someurl.apps.sustainabledelivery.com

→ Setting admin credentials on Dokku...
→ Deploying (git push dokku)...
Everything up-to-date
→ Deploy complete.
→ Waiting for app to be ready...
→ App is ready.
→ Logging in as <email> 

And then logging in, it failed. I had made the code to clever. Or rather, learnt something about which code is and is not available in the production deployment.