Data Processing in Go

Go is a good language for data processing for one reason above all others: concurrency is cheap and boring. There is no thread pool to configure, no callback pyramid to descend, and no shared-memory race to fear if you keep the pieces separate. What remains is a set of decisions about where to stream, where to batch, and where the memory limit actually is.

1. Stream Instead of Load

The first rule is that a file that does not fit in memory must never be read into memory. Go’s bufio package gives you a scanner that reads incrementally, with a sensible buffer size and no surprises.

func process(path string, fn func(line string) error) error {
    file, err := os.Open(path)
    if err != nil {
        return fmt.Errorf("open %s: %w", path, err)
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024) // allow long lines

    lineNo := 0
    for scanner.Scan() {
        lineNo++
        if err := fn(scanner.Text()); err != nil {
            return fmt.Errorf("line %d: %w", lineNo, err)
        }
    }
    return scanner.Err()
}

Two details are routinely missed. scanner.Scan() returns false at end of file or on error, so you must check scanner.Err() afterwards - skipping it silently truncates the file. And Scanner has a default token limit of 64 KB, so a file with a long JSON line fails with “token too long” unless you raise the buffer.

For CSV, encoding/csv handles quoting and escaping that strings.Split gets wrong.

reader := csv.NewReader(file)
reader.FieldsPerRecord = -1 // allow ragged rows
reader.ReuseRecord = true     // reuse the slice between rows

for {
    record, err := reader.Read()
    if errors.Is(err, io.EOF) {
        break
    }
    if err != nil {
        return fmt.Errorf("read csv: %w", err)
    }
    handle(record)
}

ReuseRecord = true avoids one allocation per row, which is the difference between comfortable and painful on a ten-million-row file. The trade-off is that the slice is overwritten on the next call, so copy anything you intend to keep.

2. Orchestrating Concurrent Work

A worker pool gives you bounded parallelism: a fixed number of workers pulling from a channel of jobs.

func processAll(ctx context.Context, jobs <-chan Job, workers int) error {
    var wg sync.WaitGroup
    errCh := make(chan error, workers)

    for i := 0; i < workers; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for job := range jobs {
                if err := job.Run(ctx); err != nil {
                    errCh <- fmt.Errorf("job %s: %w", job.ID, err)
                    return
                }
            }
        }()
    }

    wg.Wait()
    close(errCh)

    var errs []error
    for err := range errCh {
        errs = append(errs, err)
    }
    return errors.Join(errs...)
}

The number of workers should usually be tied to what the work is bound by. If the job is CPU-bound, runtime.NumCPU() is the right starting point; if it is bound by a network call or a database, the number is set by the server’s capacity, not by your machine’s core count, and going higher just moves the queue.

workers := runtime.NumCPU()
if cfg.MaxConcurrency > 0 {
    workers = cfg.MaxConcurrency
}

A sync.Pool reuses allocated objects across workers and cuts the garbage collector’s work substantially in allocation-heavy loops.

var bufPool = sync.Pool{
    New: func() any { return new(bytes.Buffer) },
}

buf := bufPool.Get().(*bytes.Buffer)
buf.Reset()
defer bufPool.Put(buf)

3. Batch Writes

Writing one row at a time is the classic mistake: each write is a round trip, and the throughput collapses to the network latency divided by the batch size.

const batchSize = 500

func writeAll(ctx context.Context, db *sql.DB, rows []Record) error {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return fmt.Errorf("begin: %w", err)
    }
    defer tx.Rollback()

    stmt, err := tx.PrepareContext(ctx, `INSERT INTO records (id, name, value) VALUES (?, ?, ?)`)
    if err != nil {
        return fmt.Errorf("prepare: %w", err)
    }
    defer stmt.Close()

    for i, r := range rows {
        if _, err := stmt.ExecContext(ctx, r.ID, r.Name, r.Value); err != nil {
            return fmt.Errorf("insert %s: %w", r.ID, err)
        }
        if (i+1)%batchSize == 0 {
            if err := tx.Commit(); err != nil {
                return fmt.Errorf("commit: %w", err)
            }
            tx, err = db.BeginTx(ctx, nil)
            if err != nil {
                return fmt.Errorf("begin: %w", err)
            }
            stmt, err = tx.PrepareContext(ctx, `INSERT INTO records (id, name, value) VALUES (?, ?, ?)`)
            if err != nil {
                return fmt.Errorf("prepare: %w", err)
            }
        }
    }
    return tx.Commit()
}

A prepared statement is reused for the whole batch, which lets the driver skip the parse step on every execution. A transaction that covers many statements turns many round trips into one commit.

The bulk-insert path is faster still when the database supports it:

// PostgreSQL: copy in one round trip
_, err := conn.ExecContext(ctx, `COPY records (id, name, value) FROM STDIN`)

4. Knowing Where the Memory Goes

At large data volumes, the question is no longer “how fast” but “what fits in memory”. A few measurements answer it.

var stats runtime.MemStats
runtime.ReadMemStats(&stats)
fmt.Printf("heap alloc: %d MiB, total alloc: %d MiB, gc cycles: %d\n",
    stats.HeapAlloc>>20, stats.TotalAlloc>>20, stats.NumGC)

pprof turns that into a picture. Expose the profile endpoints on a separate port, behind your internal network:

mux.HandleFunc("/debug/pprof/", pprof.Index)
mux.HandleFunc("/debug/pprof/heap", pprof.Index)
mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
mux.HandleFunc("/debug/pprof/goroutine", pprof.Index)

Then go tool pprof http://localhost:6060/debug/pprof/heap gives you the allocation sites, and go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30 gives you the CPU profile. Almost always the answer to “this is slow” is one of the two, and rarely is it what you would have guessed.

The usual culprits, in order of how often they turn up:

  • Slicing a large slice without a limit, which retains the whole backing array.
  • Building a map of every record when only an aggregate is needed.
  • String concatenation in a loop instead of strings.Builder.
  • Reading a whole file with io.ReadAll because the convenience was worth more than the memory.

5. What to Take Away

  • Stream with bufio.Scanner or encoding/csv, and check Err() afterwards.
  • Bound your concurrency, and size the worker pool by what the work is bound by.
  • Batch your writes, reuse prepared statements, and wrap a batch in one transaction.
  • Measure with runtime.MemStats and pprof before optimising.
  • Watch for the four usual memory traps: unbounded slices, over-large maps, string concatenation in loops, and io.ReadAll.

Practice

GoData ProcessingStreamingWorker PoolBatch WriteProfiling

Approach: The question gives a symptom (throughput flat, memory climbing, and it gets worse with more workers) and asks for the cause, so reason about the interaction between the three: how much data is held, how much work is in flight at once, and how often the results are written.

Steps:

  1. Memory climbing with a flat throughput profile points to accumulation: every record is being retained, whether in a map, a slice, or an unclosed reader. The scanner’s Err() is not being checked, or the whole file is being read with io.ReadAll, or a slice is being appended to without limit.
  2. More workers making things worse is a strong signal: if the work is bound by a downstream resource - a database connection pool, an API rate limit - then adding workers does not add throughput, it adds contention and makes each call slower. The pool size must be bounded by the dependency, not by the core count.
  3. Flat throughput regardless of batch size means the write is not actually batched: each insert is its own round trip, so the cost is latency times the number of rows. Preparing the statement once and wrapping a batch in a single transaction removes that per-row round trip.
  4. The measurement order matters: profile the CPU first to see where time is spent, then look at the allocation profile to see what is retained. Guessing from the symptom alone is unreliable, because the same flat curve can come from an unclosed file handle, a saturated connection pool, or a lock held across a network call.
  5. The fix that survives being asked for twice is the boring one: stream the input, bound the workers, batch the writes, and then look at the profile again rather than at the code.

Answer: The likely cause is that the data is accumulated in memory while the work is dispatched without a bound and the writes are not batched, so the throughput is capped by per-row round trips and the extra workers only add contention on the downstream dependency. Stream the input with a bounded reader, size the worker pool by the dependency’s capacity, batch the writes inside a transaction with a reused prepared statement, and confirm the diagnosis with a CPU and heap profile before and after the change.

More courses

Recommended courses

Found an issue with this content?

Tell us what could be improved.