Compare commits
9
Commits
17d7ad760d
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f522895e4e | ||
|
|
3308299d3c | ||
|
|
4fa0f85847 | ||
|
|
a426a2b61b | ||
|
|
d6dddbc9fb | ||
|
|
20de33720a | ||
|
|
f22a8cbf37 | ||
|
|
d7c6bea7fa | ||
|
|
419abc6b92 |
@@ -6,7 +6,7 @@ Posts live as markdown files in git and compile into the BEAM at build time: zer
|
||||
|
||||
The project structure is modular, I wanted to have a personal blog, and an easy way to add an 'engineering' blog to fledgling SaaS products, as well as release notes. The release notes now also serve to announce new features on my blog.
|
||||
|
||||
Status: Expirimental. Works on my server ;-).
|
||||
Status: Experimental. Works on my server ;-).
|
||||
|
||||
Enjoy, [Willem van den Ende](https://willemvandenende.com)
|
||||
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
%{
|
||||
title: "Swim the Test",
|
||||
author: "Willem van den Ende",
|
||||
tags: ~w(ATDD ComprehensionMaxxing CollaborationMaxxing),
|
||||
description: "Every application has some kind of workflow. Understanding larger workflows, making them cohesive and understandable to users and developers is non-trivial. I re-discovered Ward Cunningham's Swim the Test approach, this is the Further Reading section of future posts.",
|
||||
published: true
|
||||
}
|
||||
---
|
||||
|
||||
TLDR;
|
||||
----
|
||||
Every application has some kind of workflow. Understanding larger workflows, making them cohesive and understandable to users and developers is non-trivial. I re-discovered Ward Cunningham's Swim the Test approach, this is the Further Reading section of future posts.
|
||||
|
||||
[Jon Uddell's 2008 post about Ward Cunninghams' implementation of Brian Marick's 'Visible Workings'] https://blog.jonudell.net/2008/03/04/ward-cunninghams-visible-workings/ is a succinct introduction . connecting the Visible Workings pattern to the implementation made for a workflow system on behalf of the Eclipse Foundation.
|
||||
|
||||
[The "Swim" System for User-Oriented Presentation for Test-Case results](https://wiki.eclipse.org/images/2/20/Swim_System_PNSQC_2007_Paper.pdf). Plain spoken Paper by Ward Cunningham, Bjorn Freeman-Benson and Karl Matthias.
|
||||
|
||||
> Perhaps the highest compliment we received was when one of our customers (SC) said that she
|
||||
> easily understood how everything worked because it "uses her words" in the interface panels and
|
||||
> documentation.
|
||||
|
||||
Why this post?
|
||||
====
|
||||
|
||||
Refactoring integration tests towards *Swim the Test* turns out to be easier than to write about it. And was not as hard as I thought - it can be done in baby steps. I keep both losing the literature references, and at the same time finding more or them. So getting them out of the way should help me post a 'how' post more easily.
|
||||
|
||||
Swim the test
|
||||
====
|
||||
|
||||
> Finally you can *swim the test*. Here the steps and results are laid out in a
|
||||
> table. Time advances as you move down the rows, and there’s a column for every
|
||||
actor in the workflow.
|
||||
|
||||

|
||||
Screenshot by Jon Udell.
|
||||
[Jon Uddell's post]( https://blog.jonudell.net/2008/03/04/ward-cunninghams-visible-workings/) is a succinct introduction . connecting the Visible Workings pattern to the implementation made for a workflow system on behalf of the Eclipse Foundation.
|
||||
|
||||
|
||||
The "Swim" paper
|
||||
====
|
||||
|
||||
I learnt about the Swim system when Ward Cunningham ran a session about it at the first Agile Open Northwest (2006 or 2007 I think - this is a blog post, I may do the research later or it doesn't ship..)
|
||||
|
||||
[The "Swim" System for User-Oriented Presentation for Test-Case results](https://wiki.eclipse.org/images/2/20/Swim_System_PNSQC_2007_Paper.pdf) by Ward Cunningham
|
||||
Bjorn Freeman-Benson and Karl Matthias
|
||||
|
||||
I only found this paper existed now that I went looking for it. It also explained (when you read it back to front) on how to get started. The workflow with swimlanes looks very sophisticated. It mentiones `run.php` further down, which is 'just' one flow with some screenshots.
|
||||
|
||||
The reason I was attracted to domain modelling (long before Eric Evans' 2003 DDD book) was this:
|
||||
|
||||
> Perhaps the highest compliment we received was when one of our customers (SC) said that she
|
||||
> easily understood how everything worked because it "uses her words" in the interface panels and
|
||||
> documentation.
|
||||
|
||||
Simplicity is achieved when there is nothing left to take away. Compare this to DDD's "Ubiquitous Language". The scare quotes are intentional. As a non-native speaker I had to look up Ubiquitous. Compare this with the paragraph above. I appreciate many of the user-focused things that come out of the DDD community, like Event Storming, but ask yourself, if a patterns introduce barriers to outsiders, or if it removes them (and when introducing barriers, is the trade-off worth it?).
|
||||
|
||||
>At the same time,
|
||||
> our abstractions are detailed enough to be believed because we are running the real code in our
|
||||
> simulations instead of bubbles and arrows diagrams. We respect the user's expertise without
|
||||
> dumbing down our system or forcing them to use our terminology.
|
||||
|
||||
|
||||
A large application is built in many small steps
|
||||
=====
|
||||
|
||||
This looks like a lot of work, and I haven't integrated this in an application. Yet. Reworking existing integration tests to output dialogs as HTML for each step was not _that_ much work, and the rewards are instantaneous: multi-step workflows now become glanceable. And as they get longer, I can make small improvements to improve glanceability.
|
||||
|
||||
I am trying to draw in the users as well, let's see how this goes.
|
||||
|
||||
Further, further, Reading
|
||||
====
|
||||
|
||||
This is meta - a Further Reading section in a further reading post ;-).
|
||||
|
||||
A harness for behaviour
|
||||
----
|
||||
|
||||
I was reminded of the swim system by this talk of Matteo Vaccari on youtube which focuses on tests that are understandable by the stakeholders.
|
||||
[A Harness for Behaviour: Ensuring Ai-generated Code Does What We Want - TDD in the Age of AI ](https://www.youtube.com/watch?v=3tdmoj35HG0)
|
||||
He referenced Fit and focuses on writing tests that work as a communication medium between users, stakeholders and developers. He uses an external DSL in YAML. I am not against YAML, but someoone's remark last year that all new programming languages appear to be YAML gives me pause, whenever I see it. The main thing is when it works it works. Context is key. (not further reading: I ended up with an internal DSL and screenshots in small steps as a medium - the users don't need to be able to write tests, yet, communication about what we are building and how it works is key. In small steps - I don't believe good UX comes from large batches). [LinkedIn post on Matteo's account](https://www.linkedin.com/posts/matteovaccari_the-video-of-my-presentation-at-platmosphere-share-7478330775579586560-vSoX) in case you want to comment.
|
||||
|
||||
Matteo blogged about this earlier:
|
||||
[Acceptance tests for AI Assisted development](https://matteo.vaccari.name/posts/acceptance-tests-for-ai-assisted-development/) on "grey box testing". Refers to [Approved Fixtures Pattern](https://lexler.github.io/augmented-coding-patterns/patterns/approved-scenarios/) and is also worth reading on its' own.
|
||||
|
||||
Visible Workings
|
||||
---
|
||||
|
||||
[Visible Workings]( https://web.archive.org/web/20110202132102/http://visibleworkings.com) on archive.org, as the original site is no longer active. I like the notion of "Tinkerable Software".
|
||||
|
||||
>Explanatory software goes hand in hand with tinkerable software because the dedicated amateur will need explanations to do the tinkering well.
|
||||
|
||||
|
||||
"View Source"
|
||||
----
|
||||
|
||||
A comment in one of Jon Udell's posts about 'View Source' reminded me of the work of [Sue Smith](https://www.sue.codes/) My timebox is up, she has a great blog in which is she writes about building tools in an IDE to make software development accessible to newcomers. I remember her writing about how to replicate the 'view source' experience of early web 2.0, where you could find out how a web front-end worked by 'just' viewing the source.
|
||||
|
||||
[What comes after view source?](https://www.sue.codes/blog/afterviewsource/)
|
||||
|
||||
[Being intentional about friction](https://www.sue.codes/blog/whatfriction/) is more tangential, and inspiring. Some comment about the link to the tests in an application on one of Jon Udell's 'swim' posts may be related. That link to the tests inside an application dialog may be seen as friction, but also, as Sue puts it:
|
||||
|
||||
> Removing friction does not necessarily empower people
|
||||
|
||||
and
|
||||
|
||||
> Being able to choose which details are worth paying attention to is the trick.
|
||||
|
||||
Happy reading and watching
|
||||
====
|
||||
|
||||
Thank you for making it to the end. I hope this inspires you to do more reading, watching, and more importantly, building.
|
||||
|
||||
As [Steve Freeman put it just now](https://mastodonapp.uk/@sf105/116991301858959311):
|
||||
|
||||
> ward and Bjorn’s “swimlanes” work for the Eclipse foundation was an inspiration. It’s a crime against software that it was left derelict.
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
%{
|
||||
title: "Learning to Swim test",
|
||||
author: "Willem van den Ende",
|
||||
tags: ~w(ATDD ComprehensionMaxxing CollaborationMaxxing SyntheticTDD Refactoring),
|
||||
description: "The faster you can build, the more pressing communication with others and understanding together becomes. You can build software to promote understanding and collaboration, but where do you start? I found a foothold in the 'swim tests' paper and existing integration tests.",
|
||||
published: false
|
||||
}
|
||||
---
|
||||
|
||||
I often find it easier to do and build stuff than to write about it. Getting over the 'further reading' section in [swim the test](https://willemvandenende.com/blog/engineering/swim-the-test) appears to have freed me up to write more.
|
||||
|
||||
I wrote with the LinkedIn comment announcing that post:
|
||||
|
||||
> The faster you can build, the more pressing communication with others and understanding together becomes.
|
||||
|
||||
A view of a foothold
|
||||
====
|
||||
|
||||
Reading the "Swim System" paper, it is clear that you can build software to promote understanding and collaboration, but where do you start? I found a foothold in the existing integration tests, and a discussion on page 14 and 15 of the [Swim Systems paper](https://wiki.eclipse.org/images/2/20/Swim_System_PNSQC_2007_Paper.pdf).
|
||||
|
||||

|
||||
|
||||
The left part of the screenshot, a dialog with some steps, was something I could build. There is a description of how they abstracted the report creation so one source produces multiple targets, e.g. the dialog on the left and the swimlane view on the right. That is also something that can be factored out later, starting with a point solution.
|
||||
|
||||
I eventually read the whole paper front to back after skimming it and pondering it over. I also remember what Ward Cunningham mentioned when he presented this work: that tools like Fit(Nesse) and this are contextual, and these are something to be made in context, not off-the-shelf tools. So I read this paper, contrasted it with Matteo Vaccari's work, and my own experience in stakeholder and user-focused tests, and what we needed for the application I am working on.
|
||||
|
||||
What do we want?
|
||||
====
|
||||
|
||||
From the swim systems paper
|
||||
---
|
||||
|
||||
I liked:
|
||||
|
||||
- application dialogs as a way to explain the system
|
||||
- backlinks to the tests from the dialogs in the actual application
|
||||
- display of errors inline
|
||||
- The swimlanes, and the way events are used
|
||||
- Standard flows and variations on them, implemented and inspectable simply
|
||||
|
||||
I think the paper predates the Event Sourcing hype, seeing the past-tense framed things in the swimlanes
|
||||
|
||||
What I missed from the paper, and can now see after starting to build:
|
||||
|
||||
- Failing tests also displayed in the report (see the red "Trouble can't find <x>" sections on the right)
|
||||
|
||||

|
||||
|
||||
What I wanted to do differently:
|
||||
|
||||
Some actions are abstract enough, eg. _login 'developer1'_ others are too UI focused for what I need. The order in which fields are filled in is often not important. Or so I thought, when writing this I have written some tests where one action makes another one (im)possible. So most of the time a whole dialog can be worked through by data, using the Test Data Builder pattern to allow for differences.
|
||||
|
||||
I have a test setup out of the box (Phoenix Liveview Test) that works well enough, and I guessed (correctly) that it would have primitives to export HTML for the reports as well.
|
||||
|
||||
So to start with, I decided to produce reports, and leave reporting on unmet expectations to regular assertions in the existing test framework.
|
||||
|
||||
From Matteo Vaccari's ATDD and AI talk
|
||||
----
|
||||
|
||||
Matteo presented an external DSL. I liked the idea of a DSL, and decided to start with an internal DSL first. Elixir has good support for this, and I could drive this by extracting helpers from existing tests and making the scenarios longer so there are more steps in the workflow.
|
||||
|
||||
Some of the lines in Matteo's DSL I found a bit HTTP specific on seeing it, but then found that my tests made by [Synthetic TDD](/blog/engineering/synthetic-tdd) also suffered from this. At one point a response is parsed for an identifier that is used in subsequent tests.
|
||||
|
||||
See below in working in small steps on how I am dealing with that.
|
||||
|
||||
From my own failures
|
||||
----
|
||||
|
||||
- being able to see the whole and have confidence it fits together coherently
|
||||
- testing as a feedback mechanism was secondary, strangely enough
|
||||
|
||||
Having the screens, step by step, gave me confidence where I was, and made it easy to identify the next step. It did turn out to have value as a technical feedback mechanism earlier than expected. I had worked bottom up for part of a feature, and added the next step in a flow once the parts were done. Some bits to integrate them were missing, as evidenced by the new step. Good encourgement to work acceptance test driven the next time.
|
||||
|
||||
|
||||
On being able to have confidence in coherence
|
||||
---
|
||||
|
||||
The swim systems paper appealed to me, because I have trouble keeping an overview of workflows with multiple steps. This was always a challenge working in the XP way, as we very quickly can create a well-tested app with complicated flows. Keeping the whole coherent and "ready to mind" is hard. "Ready to hand" with good unit tests became very cheap, but seeing the forest and the trees was sometimes difficult.
|
||||
|
||||
This became worse with LLM generated systems. When I can create a prototype out of thin air in a couple of hours, with hundreds of tests, and several screens and can let a coding agent produce the demos (see previous posts) it feels great. But when someone else is interested and asks me how it fits together, that becomes a lot harder.
|
||||
|
||||
I was hesitant to take some of my visual collaboration tools to market, and I did not understand why. After building the first steps of workflow tests I do understand. It may look like a collaboration tool is about the sticky notes, but the magic is in the collaboration rules that are encoded in things like how groups are structured, how invitations work, what the unit of a 'tenant' is in a SaaS applications, who gets to (dis)invite who, how, when? Or in moderation: who invited this _Person_ and why?
|
||||
|
||||
It is great brainstorm-building a mobile first product outcomes focused board, but then adding teams and seeing bits of flow missing is annoying. These flows are necessary for any collaborative application, visual or not, and they are different, because the combination of peoples' roles and assumptions are unique for each application.
|
||||
|
||||
So I wanted, just for myself, to see as much of the flow as possible. End-to-End tests through browsers are now very cheap to make (I can make dozens in a day), but maintaining and running them is not. Especially when I am working on my laptop with a coding agent. Having sandboxed the agent, accessing a browser is not trivial.
|
||||
|
||||
The unit and integration tests I have so far do give me confidence that most things work, but not that the whole application flows hang together as I want. This was triggered by a stakeholder asking: "Where are we with <tool> 2.0?" . (more about <tool> later. previous blog posts drafts failed on digressions).
|
||||
|
||||
Every application has workflows. They may not appear to be core, but they shape collaboration, so they are.
|
||||
|
||||
|
||||
Working in small steps, AI augmented where useful
|
||||
=====
|
||||
|
||||
I had an integration test that could serve as a starting point. But the workflow only had two steps. Several things are taken care of beforehand, analogous to the `login(''developer1')` in the Swim Systems paper. At least it was a good place to start, so the process was roughly:
|
||||
|
||||
1. Find out how to write pages and dialogs (a part of the page) as html
|
||||
2. Decide how to collect these as the test progresses
|
||||
3. Turn the collected parts into a report
|
||||
4. Refactor the existing test towards a DSL
|
||||
5. Add more steps
|
||||
6. Go to 4.
|
||||
|
||||
For 1. I found a forum post, and I already had some [Synthetic](/blog/engineering/2026/07-13-synthetic-tdd.md) tests. This was also a good opportunity to re-read [the documentation](https://phoenix-live-view.hexdocs.pm/Phoenix.LiveViewTest.html). Rendering pages, components, selecting elements and getting the text back is all built in, so all we need to do is wrap it in a page and save the parts. I did consider writing the reports out as markdown, with html snippets at some point. When rendering html to a pdf, the page breaks happen in the middle of screenshots sometimes. But the PDF already lacks the styling.
|
||||
|
||||
I initially iterated with [Pi](/blog/engineering/tag/pi.dev) on how to collect tests. I had a fancy idea of collecting the various dialogs in a process (well supported e.g. by Elixir GenServers), then thought of doing it the unix way (write out dialogs, than `cat` them all together), and ended up collecting step outputs in a list, and rendering the list at the end. This did require re-ordering the test a bit: the `assert` has to come at the end, after creating the report.
|
||||
|
||||
My assumption was that a model could generate more steps, after refactoring towards a simple flow of steps. The generated tests had some noise in parsing outputs to determine inputs for the next steps. Finding the right extractions was a bit of work, and at the same time gave me a better understanding of what I wanted.
|
||||
|
||||
Away from the desk, away from the screen?
|
||||
----
|
||||
|
||||
I was mostly away from my desk last week. After getting three steps working, in small steps, refactoring by hand (my local model had some trouble moving things around in to files, and I needed to learn more about how to organise tests and support code in a Phoenix Liveview project ), the basic flow is now at five steps. Pi with Deepseek v4 flash did most of that, with me doing 'yes' prompts from my phone.
|
||||
|
||||

|
||||
|
||||
This insufficient credits message hasn't happened to me in a while. I deposited 25 USD in may last year on OpenRouter. Since then I mostly used Claude Code or local models. Being away from my desk means running Pi on a server, with deepseek. I spent about 10$ in the last month, preparing a keynote and TDD with AI workshops for Philips (hit me up if you are interested too, at QWAN we are responsibly extending our offerings, now that we have a better understanding of how to augment teams, and improve the outcomes and quality using coding agents). So probably about 1 or 2 $ on workshop preparation and 8 for the testing support and bringing a vibe TDD'ed app to production.
|
||||
|
||||
It is nice to be able to make progress while having friends over. I have learned to not multi-task too much, and when a question is too big to answer on my phone, I will just percolate on it. Good designs come from working on that - you have to be seen working for creativity to happen, and from stepping away from the screen and coming back with a fresh perspective.
|
||||
|
||||
|
||||
Show me the reports
|
||||
====
|
||||
|
||||
TODO add validation failure report, and full flow so far.
|
||||
|
||||
Reworking is easier than writing
|
||||
====
|
||||
|
||||
Reworking and refactoring existing user-focusing integration tests towards this turns out to be easier than writing about it. A number of ideas collide, and I had trouble keeping track of the references. So the first blogpost about this was the "further reading" section.
|
||||
|
||||
And here as well, Programming as theory building works. I built towards the principles above, learnt a few things, got some value out early, and now have an idea of the next steps.
|
||||
|
||||
|
||||
|
||||
Further Reading
|
||||
----
|
||||
[Swim the test](https://willemvandenende.com/blog/engineering/swim-the-test) has more of the literature background, including the Sim Systems paper and what I mentioned about Matteo Vaccari.
|
||||
|
||||
The Swim Systems paper is well worth reading, it is as good as any of my favourite software engineering papers from the 1970s (but 30 years more recent).
|
||||
|
||||
Felienne on Programming as Theory Building
|
||||
|
||||
Refactoring to Patterns
|
||||
|
||||
John Nolan on explaining what happened after the fact
|
||||
|
||||
Machiel Groeneveld on wanting the reasoning
|
||||
@@ -0,0 +1,25 @@
|
||||
%{
|
||||
title: "Retort",
|
||||
author: "Willem van den Ende",
|
||||
tags: ~w(Synthetic TDD pi.dev coding-agents evals ),
|
||||
description: "At first I was bemused by Retort, Adrian Cockroft's whole stack coding agent evaluation tool. It is growing on me, and might be useful as a starting point for custom evals.",
|
||||
published: true
|
||||
}
|
||||
---
|
||||
|
||||
I was bemused by Adrian Cockrofts' [Retort](https://github.com/adrianco/retort) - it looks like Claude Code mostly independently running evals on coding agent x model x target language combinations. The most recent report however, matches my anecdata. I have started on something like Retort earlier this year, but much smaller - custom tiny agent, a couple of small language models and about 10 programming languages with a number of small tasks (extract method, find code smells in a file etc). I parked it, because it looked like that would be significant work to turn into something explainable to more people. So it is great to see Retort blossoming.
|
||||
|
||||
The main takeaway for me from the latest one is that the number of turns taken by the agent seems to be a good predictor for speed of task completion, and Qwen 35B, that I use a lot, does as well as the better frontier models on this task (and better than some of the worse frontier models). My hypothesis (hope?) that a decent harness with a smaller model would perform well on incremental tasks seems to pan out.
|
||||
|
||||
Reading the readme of retort, the factors are free to label and create. It produces a number of synthetic 'blogs'. The [optimal blog](https://github.com/adrianco/retort/blob/main/optimal-blog.md) I find interesting, bedcause it compares anthropic frontier models with one model I use regularly (Qwen3.6-35B-A3B) and one that I have used successfully for legacy code analysis (Qwen3-Coder-Next 80B ).
|
||||
|
||||
Not all outcomes match my experience, Qwen3.6-35B-A3B does well enough with Elixir production code, although it struggles with macros and moving functions across modules. That, admittedly, is something more LLMs struggle with. Explaining that often takes more time than doing it by hand.
|
||||
|
||||
It would be interesting to see the same benchmarks with Deepseek V4 Flash, widely used on somewhat larger machines (128GB VRAM, or less VRAM + more RAM combinations) and cheap on Openrouter (I haven't gone over a dollar yet for a day of intensive use), and Qwen 3.6 27B (much slower in tokens per second on my Macbook pro M3 Max, but seems to get more complex tasks done than 35B. I tried it yesterday on a Docker deploy that 35B struggled with, and it fixed the two issues I had independently. Feedback loops do work, as long as the model is up to the task ).
|
||||
|
||||
Reading through the _optimal_ blog I found the source of my bemusement. Retort also does some BDD / TDD / ATDD and 'neutral' (no tests?) comparisons. I believe these are all [Synthetic TDD](https://willemvandenende.com/blog/engineering/synthetic-tdd). I have used coding agents (mainly Claude Code up to april) extensively to generate tests, recently with the help of Allium. Without tests coding agents go off the rails quickly. And I am lazy, as a good programmer should be, so I like having tests generated and providing feedback loops. But synthetic tests do have limitations. They carry an LLMs interpretation of your intent, and may not drive the design.
|
||||
|
||||
It may be that design doesn't matter everywhere equally. I am now working on some code where it does, and explaining myself in code (with some assistance) is faster and more precise than talking to an LLM.
|
||||
|
||||
I am not sure how that kind of intent could fit in a benchmark like Retort. What do you believe? Does intent and design in tests matter? If so, where and when?
|
||||
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
%{
|
||||
title: "Developing with open weights models only, three months in",
|
||||
author: "Willem van den Ende",
|
||||
tags: ~w(pi.dev open-weights ),
|
||||
description: "It is a bit over three months ago that my linkedin post about cancelling my Claude Max subscription went viral. Time for an update. The first six weeks were uneventful, then I started missing some things from before, and yesterday I closed the loop.",
|
||||
published: true
|
||||
}
|
||||
---
|
||||
|
||||
I blogged about [My local agentic development setup](/blog/engineering/my-local-agentic-dev-setup-today) the day after my post about cancelling my Claude Max subscription and going all in on local models went viral.
|
||||
|
||||
I am still using the same flexible harness - [Pi.dev](/blog/engineering/tag/pi.dev), with surprisingly few extensions (apart from yesterday, see below for prime-agent). I had expected to add web-search, but even my local models have a lot of knowledge, know how to use tools and look up urls from documentation. Enough to start building [Swim the test](/blog/engineering/swim-the-test), which is quite niche, into an existing system. I occasionally paste a hand-searched link into the prompt and that is enough.
|
||||
|
||||
The first six weeks were uneventful. I didn't need to create large prototypes in a day like before, that helped. I was busy preparing larger sessions (a coding dojo with 70 participants and 10 facilitators for a UK client, and a keynote about AI and TDD and a workshops on AI with TDD for Philips Image Guided Therapy). So mostly smaller things, and sometimes making small in-the-loop changes to existing code.
|
||||
|
||||
Then in a conversation with Chris Matts he mentioned he sketches UI on a piece of paper and then lets one of the hosted solutions (Figma or ChatGPT) create a screen out of it. I wasn't set up to do that. After that I have set up [image recognition in Pi.dev](/blog/engineering/enabling-a-local-model-to-explain-images-in-pidev). Surprisingly good. I haven't used it to generate screens, but could make that with some extensions.
|
||||
|
||||
In July I went back to working on a larger piece of software, and improvements to local inference engines were coming in thick and fast. I can now run Qwen3.6 27B, their dense model, at conversational speed, and my previous go to, their 35B mixture of experts model, runs a lot faster as well. For smaller, me-in-the-loop changes with tests, the 35B model works quite well. For trickier things, or larger independent runs, where the coding agent runs through a list of tasks and executes them in sub-agents one by one, the 27B model performs better. Slower per token, faster to outcome.
|
||||
|
||||
With [mtplx](https://github.com/youssofal/mtplx) instead of LLamaCPP I can now run the dense Qwen3.6 27B at conversational speed (was below 20 tokens per second, now often 30 at the start of a conversation). This means I can run more complex development tasks locally. If it slows down later, I'll go do something else, as long as it can finish the job independently.
|
||||
|
||||
As an example, it managed to find something properly obscure that Deepseek v4 flash also could find, but the 35B model could not. I am keeping that private as a benchmark. Before this, none of the MLX (mac accellerated) inference engines could handle long context (for me). Mtplx does that - speed does not matter if you can't get a good outcome.
|
||||
|
||||
MTPLX is also a proper consumer solution. It comes with a GUI to select models and see how it is running. It will figure out the best parameters to run a model with by running a benchmark on your machine after downloading a model, and comes with pre-selected models and quantizations. The screenshot below was after installing a smaller variation of the qwen27B model. Before this I had never seen any of them do more than 20 tokens per second. Around 30 is good enough for a conversation for me.
|
||||
|
||||
The screenshot below was taken when I installed a smaller version of the Qwen 27B model. It shows a few sessions tested on my laptop, and which one was found to be fastest, for me.
|
||||
|
||||

|
||||
|
||||
I still have LLamaCPP, but mtplx does well enough at the moment. I would us LLamaCPP when I want to run multiple smaller models on my mac simultaneously - mtplx so far only runs one model at a time. No problem for 35B and 27B as more than one does not really fit in 64GB (V)RaM.
|
||||
|
||||
Since the school holidays were starting and I am away from my desk more often, I revived my [remote environment in a virtual machine](/blog/engineering/how-to-get-started-with-the-pi-coding-agent-on-a-vps), so I could develop on my phone. I had used DeepSeek V4 Flash in part when preparing exercises. This model can be locally on slightly more VRAM than I have.I use openrouter for it. This has worked very well for adding features to existing code, doing longer running implementation sessions etc. It is cheaper to host than Qwen 3.6 27B apparently.
|
||||
|
||||
On Friday I spent a day with prime-agent. This is a ready-to-use Recursive Language Model adaptation of Pi. This runs instances of Pi on subtasks, and creates memories of things that could have been done better. It appears to do long running tasks without a ticket system like yaks, beads or a markdown file well enough. It is an example of not attaching new ways of working to old flows (ticket systems, version control). There may be better ways to discover. RLM appears to be one of them. I am doing `new session` less often, longer contexts seem to work well enough (although slower). As the memory built up, starting 'fresh' also is fast enough also is fast enough
|
||||
|
||||
[Prime-agent](https://github.com/PrimeIntellect-ai/prime-agent) means I can do what I did with claude code before, developing larger prototypes with tests in a conversation. But now with smaller open weights models. The 0731 release of deepseek-v4-flash appears to be strong. Prime-agent with feels more coherent than claude code in april. After three months of not doing larger prototypes, and developing all the things I had thought of before, I now have some new ideas. The harness is at least important as the model, and experiments in harness development are ongoing by many people.
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 187 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 458 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1014 KiB |
Reference in New Issue
Block a user