Software Engineer with a passion for mobile and web development.
While building RagPack, I needed a file worker that would parse file chunks and write them into LanceDB, an embedded columnar vector database. This would have been fiddly to get right in most languages, but Go’s concurrency primitives made it simple without any major architectural shift. Here’s how goroutines, channels, and sync.WaitGroup work together in the ingestion pipeline.
While setting up CI for LanceDB Explorer, a VS Code extension I built for browsing LanceDB databases, I wanted git tag vX.Y.Z && git push to publish straight to the VS Code Marketplace.
Got an email from Vercel about reaching limits and checked to see Fluid Active CPU spiking for worldcuppicks.co. Obviously I don’t want to go to premium, so I implemented certain measures to bring it down. We are in the knockout rounds now, so I wanted to get it sorted before things got worse.
More …While building RagPack, a library that chunks files for embedding, I needed a common way to stream parsed content from multiple file formats. RagPack supports CSV, PDF, DOCX, HTML, XLSX, Markdown, JSON and more. Each format has its own parser, but the ingester that consumes them should not care which one it is talking to. I needed a shared contract. In Java I would have reached for an Iterator<T> or an InputStream, but in Go the answer turned out to be the iter package, introduced in Go 1.23.
When setting up CI/CD for RagPack, I wanted to publish to npm from GitHub Actions without storing a long-lived token as a secret. Tokens are annoying: they expire, need rotating, and if they leak you have a problem.
More …