Merge branch 'main' of ssh://gitea.apps.sustainabledelivery.com:3022/mostalive/firehose

2026-10-21 14:53:50 +01:00
parent aef57c54f9 9050329685
commit 5a9106b61e
7 changed files with 43 additions and 15 deletions
@@ -269,7 +269,8 @@ Further reading
 - [How to get started with the Pi coding agent (on a VPS)](/blog/engineering/how-to-get-started-with-the-pi-coding-agent-on-a-vps) — Setting up Pi on a VPS is easier than I thought.
 - [Smaller open LLMs now work for open agents](/blog/engineering/smaller-open-llms-now-work-for-open-agents) — A phase shift in quality and speed of open weight models and inference.

- Nate B Jones had a good podcast / video this week on Apples' play with local models for e.g. legal offices who can not get their work certified if their data leaves the office, no matter how encrypted it is. [Nate B Jones on Apple and the next trillion dollars](https://podcasts.apple.com/gb/podcast/ai-news-strategy-daily-with-nate-b-jones/id1877109372?i=1000763732500)0
+- Nate B Jones had a good podcast / video this week on Apples' play with local models for e.g. legal offices who can not get their work certified if their data leaves the office, no matter how encrypted it is. [Nate B Jones on Apple and the next trillion dollars](https://podcasts.apple.com/gb/podcast/ai-news-strategy-daily-with-nate-b-jones/id1877109372?i=1000763732500)
+- If you want to try this out, at least one reader had success with the [Will it Blend?](/blog/engineering/will-it-blend) approach. Point your existing coding agent with hosted models at this blog post, and ask it to set Pi and Qwen up on your machine. Best let it ask you some questions on what you want to do, and look into your hardware. This is different for mac or for Linux (I would avoid Windows for this, if possible). NVIDIA is easier than AMD for GPU. I do have this working on AMD, with [Lemonade Server](https://lemonade-server.ai), but I do not know if agents will find that by themselves.

 Full disclosure: I am long APPL, NVDA and BABA (AliBaba, makers of QWEN). I use other hardware and models too and don't have much Nvidia hardware.

@@ -282,7 +283,7 @@ What was remarkable to me in the Field Report above is that the writer chose Ope

 > Why don't I just use [..] Claude Code? I had problems due to a lack of optimization for small context windows. Long-running tasks that complete large projects independently matter for me, so no Claude Code. 

-So maybe my intuition is right. I am not doing long running tasks like I was doing with Cladue Code _yet_ with Pi and Qwen. This shows it is possible, and might even be better. Detailed prompts and deterministic harnesses make the difference between frontier models and harnesses smaller. More about that maybe later. I hope this helps you, let me know if you have any questions or remarks. 
+So maybe my intuition is right. I am not doing long running tasks like I was doing with Claude Code _yet_ with Pi and Qwen. This shows it is possible, and might even be better. Detailed prompts and deterministic harnesses make the difference between frontier models and harnesses smaller. More about that maybe later. I hope this helps you, let me know if you have any questions or remarks. 

 Acknowledgements
 ----
@@ -1,7 +1,7 @@
 %{
  title: "Synthetic TDD",
  author: "Willem van den Ende",
-  tags: ~w(Vocabulary AI TDD),
+  tags: ~w(Vocabulary AI TDD Synthetic-TDD),
  description: "Synthetic Test Driven Development is when you have a coding agent generate the tests and the code, and maybe also do the refactor step with an agent.",
  published: true
 }
@@ -1,7 +1,7 @@
 %{
  title: "Learning to Swim test",
  author: "Willem van den Ende",
-  tags: ~w(ATDD ComprehensionMaxxing CollaborationMaxxing SyntheticTDD Refactoring),
+  tags: ~w(ATDD ComprehensionMaxxing CollaborationMaxxing Synthetic-TDD Refactoring),
  description: "The faster you can build, the more pressing communication with others and understanding together becomes. You can build software to promote understanding and collaboration, but where do you start? I found a foothold in the 'swim tests' paper and existing integration tests.",
  published: false
 }
@@ -1,7 +1,7 @@
 %{
  title: "Retort",
  author: "Willem van den Ende",
-  tags: ~w(Synthetic TDD pi.dev coding-agents evals ),
+  tags: ~w(Synthetic-TDD pi.dev coding-agents evals ),
  description: "At first I was bemused by Retort, Adrian Cockroft's whole stack coding agent evaluation tool. It is growing on me, and might be useful as a starting point for custom evals.",
  published: true
 }
@@ -1,8 +1,8 @@
 %{
  title: "Video Transcriber - turn your videos into text with slides",
  author: "Willem van den Ende",
-  tags: ~w(writing local-models, open-weights AI agentic-engineering speech-to-text),
-  description: "In which the author uses a local text to speech model to draft blog posts, starting from talking while sketch noting.",
+  tags: ~w(writing local-models open-weights AI agentic-engineering speech-to-text),
+  description: "In which the author uses a local text-to-speech model to draft blog posts, starting from talking while sketch noting.",
  image: nil,
  published: true
 }
@@ -14,13 +14,29 @@ I attempted to write about Video Transcriber when I first used it nine months ag

 Instead of using AI to write my words, I am using AI to build me small tools to get my words out. These tools may or may not contain AI themselves. In today's episode, I used a Speech-to-Text model together with a deterministic [computer vision library](https://opencv.org/). I only want to record an image (a 'slide' in the domain of Video Transcriber) whenever I stop dragging the canvas around, or stop drawing for a bit. The vision library allows deterministic code to do that. It can be used instead of the existing functionality in Video Transcriber that creates a slide every 15 or 30 seconds.

-A regular pace does not seem to be a good fit for when I am sketching. I either sketch a lot, or talk a lot, but rarely at the same time.
+A regular pace does not seem to be a good fit for when I am sketching. I either sketch a lot, or talk a lot, but rarely at the same time. Extending Video Transcriber to take that into account was easier done than thought ;-). I am still surprised I can now do this kind of thing without deeply understanding the underlying technology.

+This post hinges on three things:
+
+- Using local AI to work with audio and video, and help me write
+- Using a coding agent to modify someone else's tool to taste
+- The challenge of publishing blog posts at the moment I have just done the thing
+
+It became a bit bigger than I imagined, and is less structured then when I was drafting. I did not plan to do stopped-motion detection, I swear! I hope you enjoy.
+
+Before we go on, a small sample of Video Transcribers' work. This one has timestamps to go with the text, it can also output whole paragraphs at once.
+
+![Transcript excerpt from a book review. Lines preceded by timestamps1 A paragraph of text, a heading that says 'Slide 12' with a timestamp. A partial view of a hand drawn scribble. ](/images/blog/2026/centola-barriers.png)
+
+You may notice it says "Santola" in the screenshot. That should be [Centola](https://www.damoncentola.com/). With a large Speech-To-Text model running on my laptop, this was about the only mistake. Phonetically they are the same, and I may need to work on my pronounciation. This is of course easily fixed by using search and replace.
+
+# Using local AI to work with audio and video, help me write

 ## What is Video Transcriber for?

 It is originally intended to make youtube videos easier to skim and refer back to. Reading text is a lot faster than listening to words. If you missed something, you can quickly glance back in text, whereas in video you have to scroll back and wait for the moment that you missed - it is only obvious in hindsight where that was.

+
 ## Different tools for different purposes

 For youtube videos I now use Gemini Notebook, because it already has the transcriptions. I do miss the visuals with it sometimes, and want to replace it with an open notebook that runs with local models when I want to. I hadn't used video transcriber in a while, but that changed last week.
@@ -35,14 +51,15 @@ I picked Video Transcriber back up to transcribe sketch notes. I sketch on my ta

 While I was drafting this blog post, trying to explain while I did not write about it nine months ago, I set my coding agent to work on a wish. A 6 and a half sketch noting video yielded 81 'slides' (the systems metaphor is  a transcriber for a presentation with slides). It would be nice if it only made a slide once I stopped dragging the infinite canvas around, or when I stopped writing for a bit. 

-I gave the [coding agent](/blog/engineering/tag/pi.dev) a brief prompt, and the video I made to draft a book review of Damon Centola's [Change: How to make big things happen](https://www.goodreads.com/book/show/53369466-change)
+I gave the [coding agent](/blog/engineering/tag/pi.dev) a brief prompt, and the video I made to draft a book review of Damon Centola's [Change: How to make big things happen](https://www.goodreads.com/book/show/53369466-change), together with output from Video Transcriber that made a slide every 15 seconds, for a total of 81 slides. With access to tools (ffmpeg, the OpenCV library, Python), and my question, the coding agent (Pi with GLM 5.3 flash in this case) did most of the work independently. It used Romily's demo video from the repository for quick tests, and also used my book review video to compare the resulting markdown. It did some iterations on it to find good settings, and explained three settings to me, which I had it save in a document for the future.

-Am I going to use Video Transcriber more now? Probably in the short term. In the long term, my flow of narrating could improve to the point that making a few screenshots does not feel like that much work anyomore, and the technical solution becomes less important. From what I have seen in my initial experiments, is that there are only a few significant images. Exporting the visuals from the source and drag them in at the appropriate place in the blog post is then not that much work. 
+After it more or less worked, I found that the text occasionally contained repeated phrases. The model had spotted this as well, and suggested to remove silence from the video before processing. It had found something that could do it, so  I let it rip. 

 ## Getting words out to the world

-I already invested in making it easier to put images in notes and blog posts, and export notes to blog posts. That is paying off already. Now the work is making it a habit to finish posts.
+I already invested in [making it easier to put images in notes and blog posts](/blog/engineering/removing-a-special-cause-of-variation-in-small-steps), and export notes to blog posts. That is paying off already; it is the kind of work that compounds. I have less friction in details, so I can focus on the stories I want to tell. Now the work is in making it a habit to finish and publish posts, as the drafts (with images :-) ) were piling up. 

+# The challenge of publishing blog posts at the moment I have just done the thing

 ## Why I didn't write about it nine months ago

@@ -50,8 +67,7 @@ So why didn't I write about Video Transriber earlier? I sometimes find it hard t

 Romily asked if I wanted to contribute something back, at the time, say a blog post or a pull request. I struggled with that. As a practice, [Will It Blend](https://willemvandenende.com/blog/engineering/will-it-blend) is great to get something to work, and the flow feels great, as it is very improvisational. Being able to quickly vibe code even a small improvement and use it is fantastic. A downside of it, is that I often can&rsquo;t easily explain what happened afterwards, not even with the session transcript, as the coding agent records it, in hand.

-
-## Looking back at what I did
+# Adopting someone else's tools nine months ago

 Looking back at the nine month old commits now, it was me using \`uv\` instead of \`pip\` - I standardized on uv as it felt easier and seems more modern and faster. But I am not a heavy python user like Romily, so I did not want to push my preferences at the time, and I couldn&rsquo;t quickly see how to have both co-exist in the README.

@@ -70,3 +86,11 @@ In the meantime, I would recommend checking  [Video Transcriber](https://github.
 See you next time!


+# Further Reading
+
+In [Will It Blend](https://willemvandenende.com/blog/engineering/will-it-blend)  I went into more detail in how I adjust other peoples' software to my needs. 
+
+Last year I wrote for QWAN on how I see my evolution of knowledge and tool use, in the age of Large Language models: [Stochastic to Deterministic](https://www.qwan.eu/2025/12/11/stochastic-to-deterministic.html). I do more with language and other kinds of models than before. But I still turn things I repeat into tools. Writing is still hard, but [removing accidental friction, like adding images to this blog](/blog/engineering/removing-a-special-cause-of-variation-in-small-steps) from my writing and publishing flow keeps paying off. 
+
+I published [my fork of Video Transcriber](https://github.com/mostalive/video-transcriber), in the spirit of "Great artists ship". It is [make it run, with tests](/blog/engineering/synthetic-tdd). I must admit, I let [my agent](/blog/engineering/tag/pi.dev) ignore the instructions to write one test at a time, red-green-refactor style. I find that interesting to do occasionally, especially as a workshop, to 'feel' a model up close, or when I guide a team through entangled legacy code. In this case, I wanted the outcome, and that goes faster with [Synthetic TDD](). Express intent, set up feedback loops, try it out, and do another round.
+