Skip to content

Go Program Out of Memory

Last week, during on duty, one user reported his program exited silently without any useful information during deployment, the process is aborted with a killed information: 

[1]    90616 killed     ./bin/server

This is an intrigue question as why the program is killed by OOM killer instead of a fatal stack overflow error.

Firstly, we can exclude the case that go runtime ends up the program as in such a case go will preserve all lived go-routine information. Then, the log is likely the log reported by the OOM killer.

Then, I tried to run the program locally, and find it consumes numerous memory:

img.png

Hence, I believed this user is falling into a endless loop, instead of endless recursive call which will be stack overflow instead of out of memory.

Dive Deep into The Problem

With the help of profile, the root cause is quickly found that too many Demo objects is created so they consume a lot of memory(because the deployment platform constraints the memory usage, it only uses around 1GB).

img.png

Hence, we could find the concrete code block, it looks well. However, the interval is 0 when running the code and leads an endless loop. 

func handler(items []Item) []Output {
    interval := getInterval()
    var demos []Demo
    for i := 0; i < len(items); i += interval {
        demos = append(demos, newDemo())
    }

    var o []Output
    for _, d := range demos {
        o = append(o, Output{d})
    }
    return o
}

Then quickly we need to ask, why it doesn't stack overflow? Instead, it could even consume more than 60GB memory in my own desktop. Why?

Stack Size

The default golang stack size is 2KB in golang 1.21, which is defined in src/runtime/stack.go 

    // The minimum size of stack used by Go code
    stackMin = 2048

It changes several times before, generally it changes because of the efficiency adjustment.

The maximum stack size is 1 GB on 64-bit, 250 MB on 32-bit, according to the comments.

Heap Size

Because I didn't dive too much about this, I don't know the maximum heap size and I believe it's decided by the OS. But here it doesn't matter because before reaching the limitation of heap itself, the process is killed by the OOM killer for trespassing the container limitation.

Stack or Heap Allocated Variables?

Not all variables declared inside a function are allocated on the stack in Go. Intuitively, it should be allocated on the stack so it will be freed once the function call stack ends. This is true in C++, Rust so variables allocated under a sub scope will be dangling after exits from their scope.

Go compiler allocates variables in the heap if compiler cannot prove the variables are not referenced after function returns. As the FAQ documents.

When possible, the Go compilers will allocate variables that are local to a function in that function's stack frame.

However, if the compiler cannot prove that the variable is not referenced after the function returns, then the compiler must allocate the variable on the garbage-collected heap to avoid dangling pointer errors.

Also, if a local variable is very large, it might make more sense to store it on the heap rather than the stack. For some large variables(>32KB), they will be allocated in the heap directly.

Answer: Why OOM Instead of Stack Overflow

Because the variables in the endless loop are referenced after function returns, the compiler will use heap to allocate those variables. As a result, it continues to consume the memory and is finally killed by the OOM killer.