Debug

Profiling Go Programs with pprof

Learn how to collect and analyze Go CPU, heap, goroutine, block, mutex, thread, and execution-trace data with pprof.

Profiling is the practice of measuring a running program to discover where it spends CPU time, allocates memory, retains heap objects, waits, or contends for locks. Go's pprof ecosystem collects this evidence and provides tools for exploring it.

Profiling replaces guesses based on source inspection with measurements from a representative workload. A reliable investigation uses the same workload before and after a targeted change, then checks both profile data and user-visible results such as throughput, latency, memory use, and correctness.

How pprof works

pprof has three connected parts:

• The Go runtime collects profile data, usually by sampling.
• Your program exposes the data through HTTP handlers or writes it to a file.
• go tool pprof loads one or more profiles and presents tables, source annotations, and call graphs.

A sampled profile records observations at intervals, such as stack locations seen while threads execute. It estimates behavior with relatively low overhead. An event-based tool records a more detailed sequence of events, often with greater data volume and cost. Go execution traces provide this complementary event timeline.

Profile types and diagnostic questions

Enabling HTTP pprof endpoints

Importing net/http/pprof registers handlers on Go's default HTTP multiplexer as an import side effect. The server must serve that multiplexer for the handlers to be reachable.

package main

import (
    "log"
    "net/http"
    _ "net/http/pprof"
)

func main() {
    // Serve diagnostics on a separately protected listener in real deployments.
    log.Fatal(http.ListenAndServe("127.0.0.1:6060", nil))
}

The principal paths below /debug/pprof/ include /profile for CPU collection, /heap, /goroutine, /block, /mutex, /threadcreate, and /trace. The index also lists profiles registered by the runtime and application.

If an application uses a custom http.ServeMux, importing the package alone does not place handlers on that custom mux. Register the handlers explicitly or serve the default mux.

Capturing profiles from a running service

First check that the expected process and listener respond:

curl http://127.0.0.1:6060/debug/pprof/

Generate representative traffic while collecting. An idle process produces little useful CPU evidence. A 30-second CPU capture can be opened directly with:

go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30

Longer captures usually provide more samples, but consume more operational resources and may increase overhead. Choose a bounded duration that covers normal behavior or the incident.

Other profiles can be opened directly:

go tool pprof http://127.0.0.1:6060/debug/pprof/heap
go tool pprof http://127.0.0.1:6060/debug/pprof/goroutine

Save data when you need repeatable analysis or comparison:

curl -o cpu.pb.gz 'http://127.0.0.1:6060/debug/pprof/profile?seconds=30'
go tool pprof cpu.pb.gz

Block and mutex profiles require runtime sampling configuration before the workload is reproduced:

runtime.SetBlockProfileRate(1)
runtime.SetMutexProfileFraction(1)

A rate of 1 requests very detailed sampling and can add substantial overhead. Use it temporarily during diagnosis, then reset or tune the rates for normal operation. The appropriate setting balances signal quality against runtime cost.

Writing profiles from a standalone Go program

For a command-line workload, use runtime/pprof directly. Start CPU profiling immediately before the measured operation and stop it afterward.

f, err := os.Create("cpu.pb.gz")
if err != nil {
    log.Fatal(err)
}
defer f.Close()

if err := pprof.StartCPUProfile(f); err != nil {
    log.Fatal(err)
}
defer pprof.StopCPUProfile()

runMeasuredWorkload()

Write a heap profile after the workload reaches the observation point:

f, err := os.Create("heap.pb.gz")
if err != nil {
    log.Fatal(err)
}
defer f.Close()

if err := pprof.WriteHeapProfile(f); err != nil {
    log.Fatal(err)
}

Alternatively, use pprof.Lookup("heap").WriteTo when you need direct control over profile output.

Interactive pprof analysis

Start an interactive session with a local file or a remote URL:

go tool pprof cpu.pb.gz
go tool pprof http://127.0.0.1:6060/debug/pprof/heap

Graph output can also be generated as png, svg, or dot, depending on the local graph-rendering tools and pprof options. For example, use web interactively or request an output format from the command line.

Interpreting profile measurements

Flat cost is cost attributed directly to a function's own execution or allocations, excluding its callees. Cumulative cost includes that function and the work performed below it in the call graph. A function can have low flat CPU cost but high cumulative cost because it calls an expensive function.

Samples are observations, not exact accounting for every instruction. A percentage is the function's share of the collected sample value. Call graph edges describe caller-to-callee relationships, and inclusive cost follows those relationships downward.

Results can be affected by sampling uncertainty, compiler inlining, optimization, build symbols, and the workload's phase. Runtime, operating-system, or system-call frames may be legitimate evidence rather than application bugs. Repeat captures and interpret them alongside latency and throughput measurements.

Heap sample types

High alloc_space can indicate allocation churn that increases garbage-collection work even when inuse_space is stable. Increasing inuse_space suggests retained memory, but verify reachability and capture points before calling it a leak.

Memory investigation workflow

1. Define a steady, repeatable workload and record baseline throughput, latency, and memory behavior.
2. Capture an allocation-oriented heap view to find code producing many bytes or objects.
3. Capture a live-memory view after the same workload reaches a controlled point.
4. Save a baseline live heap profile, reproduce the workload, and save a later profile.
5. Compare compatible profiles to isolate changes:

go tool pprof -base baseline.pb.gz updated.pb.gz

Use the diff to identify what grew or shrank, then trace references in source code: caches, queues, maps, closures, goroutines, and global structures are common retention owners. Retest under the identical workload. A profile diff is evidence of changed resource use, not by itself an explanation of why objects remain reachable.

Concurrency investigation workflow

Capture a goroutine profile when requests are delayed or the service appears stuck. Group similar stacks and look for many goroutines waiting on the same channel, lock, network operation, or worker lifecycle. Unexpectedly persistent stacks can indicate a goroutine leak.

Enable block profiling before reproducing the incident to measure time spent waiting on synchronization operations. Use mutex profiling to find locks with significant contention. Inspect ownership, lock scope, channel closure, worker shutdown, and external-call behavior in the identified paths.

When summaries do not explain scheduling order, network waits, garbage collection, or synchronization timing, use an execution trace. A trace can show a timeline of runtime events that a stack profile cannot.

Optimization and validation loop

1. Choose one bottleneck supported by profile evidence.
2. Make one focused algorithm or code change.
3. Run the identical representative workload with equivalent duration and concurrency.
4. Compare profiles and application metrics.
5. Confirm correctness, throughput, latency, and memory use before accepting the change.

Do not optimize a single profile number in isolation. A lower CPU cost in one function may simply move work elsewhere, and a faster allocation path may not reduce retained memory. Avoid premature micro-optimization and treat one capture as a hypothesis-supporting measurement rather than a final conclusion.

Troubleshooting pprof

The profiling URL returns 404

Check that net/http/pprof is imported, that the server is using the default multiplexer or has explicit registrations, and that the request reaches the intended listener and process.

CPU output mostly shows runtime, syscall, or unknown frames

The workload may be waiting on I/O or an external dependency rather than using CPU. Traffic may be too light, or the interval may be too short. Increase representative load and duration, inspect external waits separately, and analyze a build with useful symbols and source information.

Allocations are high but memory does not fall

You may be confusing allocation volume with retained memory. Compare alloc_* and inuse_* views, choose controlled observation points, and account for garbage collection and runtime memory capacity before investigating object reachability.

Block or mutex profiles contain little data

Enable the corresponding runtime sampling rate before reproducing contention. The collection period must include the problem, and the selected profile must represent the waiting operation. Use execution tracing for broader waiting behavior.

Collection affects behavior or reveals sensitive information

Restrict endpoint access, collect for bounded periods, use the lowest useful sampling detail, and store profile files as sensitive operational data. Detailed block, mutex, and trace collection can be more intrusive than ordinary sampling.

An optimization improves a profile number but not user-visible performance

The changed function may not be the end-to-end bottleneck, another bottleneck may have emerged, or the before-and-after workloads may differ. Repeat the experiment with consistent traffic and validate latency, throughput, memory, and correctness.

Further Go diagnostics

For runtime counters, see Go expvar variables. For a browser-oriented diagnostics view, see the default debug view. Use the pprof command-line endpoint only under the same access controls as the other diagnostic endpoints.