Static blog with front page summary
Goal: have a personal blog, and try out another point in the 'modular app design with elixir' space. Designing OTP systems with elixir had some interesting ideas.
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
%{
|
||||
title: "How We Test LiveView at Scale",
|
||||
author: "Carlos Rivera",
|
||||
tags: ~w(elixir liveview testing),
|
||||
description: "Our testing strategy for 200+ LiveView modules"
|
||||
}
|
||||
---
|
||||
With over 200 LiveView modules in our codebase, we needed a testing strategy
|
||||
that was both fast and reliable. Here's what we landed on.
|
||||
|
||||
## The three-layer approach
|
||||
|
||||
We test LiveViews at three levels:
|
||||
|
||||
1. **Unit tests** for the assign logic — pure functions, no rendering
|
||||
2. **Component tests** for individual function components using `render_component/2`
|
||||
3. **Integration tests** for full page flows using `live/2`
|
||||
|
||||
The key insight is that most bugs live in the assign logic, not in the
|
||||
templates. By extracting assigns into pure functions, we can test the
|
||||
interesting bits without mounting a LiveView at all.
|
||||
|
||||
```elixir
|
||||
defmodule MyAppWeb.DashboardLive do
|
||||
use MyAppWeb, :live_view
|
||||
|
||||
# Pure function — easy to test
|
||||
def compute_metrics(raw_data, date_range) do
|
||||
raw_data
|
||||
|> Enum.filter(&in_range?(&1, date_range))
|
||||
|> Enum.group_by(& &1.category)
|
||||
|> Enum.map(fn {cat, items} ->
|
||||
%{category: cat, count: length(items), total: Enum.sum_by(items, & &1.value)}
|
||||
end)
|
||||
end
|
||||
end
|
||||
|
||||
# In the test file
|
||||
test "compute_metrics groups and sums correctly" do
|
||||
data = [
|
||||
%{category: "sales", value: 100, date: ~D[2026-03-01]},
|
||||
%{category: "sales", value: 200, date: ~D[2026-03-02]},
|
||||
%{category: "support", value: 50, date: ~D[2026-03-01]}
|
||||
]
|
||||
|
||||
result = DashboardLive.compute_metrics(data, {~D[2026-03-01], ~D[2026-03-31]})
|
||||
|
||||
assert [
|
||||
%{category: "sales", count: 2, total: 300},
|
||||
%{category: "support", count: 1, total: 50}
|
||||
] = Enum.sort_by(result, & &1.category)
|
||||
end
|
||||
```
|
||||
|
||||
## Speed matters
|
||||
|
||||
Our full test suite runs in under 90 seconds on CI. The secret is
|
||||
`async: true` everywhere and avoiding database writes in unit tests.
|
||||
We use `Mox` for external service boundaries and `Ecto.Adapters.SQL.Sandbox`
|
||||
only for integration tests.
|
||||
|
||||
## What we'd do differently
|
||||
|
||||
If starting over, we'd adopt property-based testing with `StreamData` earlier.
|
||||
Several production bugs would have been caught by generating edge-case assigns
|
||||
rather than hand-writing examples.
|
||||
@@ -0,0 +1,64 @@
|
||||
%{
|
||||
title: "Rebuilding Our Data Pipeline with Broadway",
|
||||
author: "Jane Doe",
|
||||
tags: ~w(elixir broadway data-engineering),
|
||||
description: "How we replaced our Kafka consumer with Broadway for 10x throughput"
|
||||
}
|
||||
---
|
||||
Last quarter we hit a wall with our homegrown Kafka consumer. Message lag was
|
||||
growing, backpressure was non-existent, and our on-call engineers were losing
|
||||
sleep. We decided to rebuild on [Broadway](https://github.com/dashbitco/broadway).
|
||||
|
||||
## Why Broadway?
|
||||
|
||||
Broadway gives us three things our old consumer lacked:
|
||||
|
||||
- **Batching** — messages are grouped before hitting the database, cutting our
|
||||
write volume by 90%
|
||||
- **Backpressure** — producers only send what consumers can handle
|
||||
- **Fault tolerance** — failed messages are retried automatically with
|
||||
configurable strategies
|
||||
|
||||
## The migration
|
||||
|
||||
We ran both pipelines in parallel for two weeks, comparing output row-by-row.
|
||||
Once we confirmed parity, we cut over with zero downtime.
|
||||
|
||||
```elixir
|
||||
defmodule MyApp.EventPipeline do
|
||||
use Broadway
|
||||
|
||||
def start_link(_opts) do
|
||||
Broadway.start_link(__MODULE__,
|
||||
name: __MODULE__,
|
||||
producer: [
|
||||
module: {BroadwayKafka.Producer, [
|
||||
hosts: [localhost: 9092],
|
||||
group_id: "my_app_events",
|
||||
topics: ["events"]
|
||||
]}
|
||||
],
|
||||
processors: [default: [concurrency: 10]],
|
||||
batchers: [default: [batch_size: 100, batch_timeout: 500]]
|
||||
)
|
||||
end
|
||||
|
||||
@impl true
|
||||
def handle_message(_, message, _) do
|
||||
message
|
||||
|> Broadway.Message.update_data(&Jason.decode!/1)
|
||||
end
|
||||
|
||||
@impl true
|
||||
def handle_batch(_, messages, _, _) do
|
||||
rows = Enum.map(messages, & &1.data)
|
||||
MyApp.Repo.insert_all("events", rows)
|
||||
messages
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
## Results
|
||||
|
||||
After the migration, our p99 processing latency dropped from 12s to 180ms and
|
||||
we haven't had a single page about consumer lag since.
|
||||
Reference in New Issue
Block a user