Asterisk memory-growth evidence: MALLOC_DEBUG, mmlog and reference counts

Define one process, workload and measurement question

Record node, operating system, kernel, architecture, exact Asterisk branch and build, package or source identity, executable, modules, process ID, start time, workload, call flow, traffic rate, recent change and synchronized observation interval. Preserve resident memory, virtual memory, cgroup or service limits, swap and host pressure as separate measurements.

Rising RSS can reflect live allocations, allocator retention, caches, mapped files, stacks or another subsystem. A high value, restart drop or dashboard graph does not by itself prove an Asterisk leak, corruption or affected call.

Choose one instrument and record its build contract

Asterisk documents MALLOC_DEBUG as a Menuselect compiler option that adds memory diagnostic commands and produces /var/log/asterisk/mmlog after a successful instrumented build. Preserve the exact build flags, install target, running binary and available CLI help instead of assuming the production binary contains the feature.

The documentation warns not to combine a MALLOC_DEBUG build with Valgrind because the instruments compete and invalidate their findings. Treat either approach as a separately approved experiment with its own baseline, overhead and stop condition.

Use allocation commands as bounded snapshots

memory show summary groups allocations, while memory show allocations lists segments and can restrict by source file or fence anomalies. Asterisk also documents at-exit listing or summaries and optional allocation backtraces. Record the exact command, time, process uptime and workload for each snapshot.

An allocation list is not a time series. Backtrace collection can be CPU intensive, and large output can consume storage. Compare like-for-like intervals in a safe environment before treating persistent growth as evidence worth escalation.

Keep reference-count debugging distinct

Reference-count logging targets astobj2 object lifetimes, not every heap allocation. Current Asterisk guidance distinguishes refdebug=yes for Asterisk 14 and later from the older REF_DEBUG compiler option, and calls for a graceful Asterisk shutdown before processing the refs file to avoid false positives.

The refs log is overwritten on each run. Preserve it before restart, process an authorized copy with the documented refcounter.py options and retain normal, leaked, invalid and skewed classifications rather than collapsing every line into a leak.

Correlate mmlog, refs, activity logs and reproduction

Asterisk asks for environment and call-flow details, normal logger evidence and the mmlog when developers suspect a leak. Align memory measurements with one narrow activity log, workload change, module state, call lifecycle and reproduction interval. Stop temporary debug and protocol logging after capture.

MALLOC_DEBUG output, reference records, Valgrind findings, ordinary logs and a repeatable reproduction answer different questions. None alone proves the leaking owner, caller impact, compatibility, a corrective patch or service restoration.

Protect sensitive artifacts and bound production impact

Asterisk warns that logs and CLI output can expose passwords and call history. Memory diagnostics may reveal filenames, functions, addresses, call-flow context and other operational data. Keep raw artifacts restricted, create a redacted issue copy and record custody, recipients and retention separately.

Instrumentation, debug verbosity, backtraces, large files and graceful shutdown can affect CPU, storage or calls. Use an approved non-production reproduction where possible; otherwise document the owner, window, abort threshold and ordinary-call validation without claiming laboratory confirmation.

Do not combine memory and deadlock findings

The deadlock-evidence guide preserves lock stacks and all-thread backtraces separately from allocation and reference-count evidence.