[SECURITY] Heap-buffer-overflow at src/mkvlib.c:44 through max_lvl allocation wrap

[shootduck]

Vulnerability description

The standalone genmkvpwd utility allocates the global nbparts memoization table in src/genmkvpwd.c and later reads it in src/mkvlib.c. The max_lvl argument is stored in an unsigned int. Its upper bound is applied only after the max_len == 0 branch, so an oversized max_lvl reaches the allocations in that branch.

max_lvl = atoi(argv[2]);

if (max_len == 0) {
    for (max_len = 6; max_len < 20; max_len++) {
        nbparts = mem_alloc(256 * (max_lvl + 1) *
                            sizeof(long long) * (max_len + 1));
        memset(nbparts, 0, 256 * (max_lvl + 1) *
               (max_len + 1) * sizeof(long long));
        nb_parts(0, 0, 0, max_lvl, max_len);
    }
}

For the supplied command, max_lvl is 16,777,216 (2^24). The first multiplication is evaluated as 32-bit unsigned arithmetic:

256 * (16,777,216 + 1) = 2^32 + 256 -> 256

At the first loop iteration, max_len is 6, so only 14,336 bytes (1,792 uint64_t elements) are allocated. The supplied stats file assigns proba1[1] = 1. The first recursive child therefore reads:

nbparts[1 + 1 * 256 + 1 * 256 * 6] = nbparts[1793]

Element 1,793 is exactly one element past the 1,792-element allocation. This is an 8-byte heap out-of-bounds read at src/mkvlib.c:44.

Version and commit

John the Ripper 1.9.0-jumbo-1+bleeding, commit 9a336d8 (built version string: 1.9.0-jumbo-1+bleeding-9a336d800a).

Environment

• Ubuntu 24.04.4 LTS; Linux 6.8.0-136-generic; x86_64.

• GCC 13.3.0, glibc 2.39, and GCC AddressSanitizer.

• The binary was configured with --without-openssl --enable-asan, debug information, and frame pointers.

Steps to reproduce

1. On Ubuntu, install the build prerequisites:

   sudo apt-get update
   sudo apt-get install -y build-essential git perl

2. Check out the vulnerable revision and set POC_STATS to the absolute path of the included stats artifact:

   export JTR=$PWD/john
   export POC_STATS=/absolute/path/to/stats
   git clone https://github.com/openwall/john.git "$JTR"
   git -C "$JTR" checkout 9a336d800a091bec9650c29282485145f31c9ffc

3. Build genmkvpwd with AddressSanitizer:

   cd "$JTR/src"
   ./configure --quiet --without-openssl --enable-asan CC=gcc \
     CFLAGS='-O0 -g -fno-omit-frame-pointer' \
     CFLAGS_EXTRA='-O0 -g -fno-omit-frame-pointer' \
     LDFLAGS='-fsanitize=address'
   make -j"$(nproc)"

4. Run the PoC. AddressSanitizer terminates the process with heap-buffer-overflow:

   cd "$JTR/run"
   ./genmkvpwd "$POC_STATS" 16777216 2> sanitizer_report.txt

Sanitizer report

The following is the complete, unmodified report included as sanitizer_report.txt.

=================================================================
==785159==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x528000003908 at pc 0x5f323c843cb8 bp 0x7ffff92f02c0 sp 0x7ffff92f02b0
READ of size 8 at 0x528000003908 thread T0
    #0 0x5f323c843cb7 in nb_parts /home/shootduck/hotracer-experiments/0-day/hotracer/pilot/genmkvpwd/2026-08-01-1.9.0-jumbo-1+bleeding-9a336d8/john/build-asan/src/mkvlib.c:44
    #1 0x5f323c843d86 in nb_parts /home/shootduck/hotracer-experiments/0-day/hotracer/pilot/genmkvpwd/2026-08-01-1.9.0-jumbo-1+bleeding-9a336d8/john/build-asan/src/mkvlib.c:49
    #2 0x5f323c84291e in main /home/shootduck/hotracer-experiments/0-day/hotracer/pilot/genmkvpwd/2026-08-01-1.9.0-jumbo-1+bleeding-9a336d8/john/build-asan/src/genmkvpwd.c:235
    #3 0x7e36a742a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #4 0x7e36a742a28a in __libc_start_main_impl ../csu/libc-start.c:360
    #5 0x5f323c840784 in _start (/home/shootduck/hotracer-experiments/0-day/hotracer/pilot/genmkvpwd/2026-08-01-1.9.0-jumbo-1+bleeding-9a336d8/john/build-asan/run/genmkvpwd+0x4784) (BuildId: 4875d0d0571be6b5aca6c0dbdb99f1729b7a84ad)

0x528000003908 is located 8 bytes after 14336-byte region [0x528000000100,0x528000003900)
allocated by thread T0 here:
    #0 0x7e36a78fd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5f323c845b27 in mem_alloc /home/shootduck/hotracer-experiments/0-day/hotracer/pilot/genmkvpwd/2026-08-01-1.9.0-jumbo-1+bleeding-9a336d8/john/build-asan/src/memory.c:92
    #2 0x5f323c842833 in main /home/shootduck/hotracer-experiments/0-day/hotracer/pilot/genmkvpwd/2026-08-01-1.9.0-jumbo-1+bleeding-9a336d8/john/build-asan/src/genmkvpwd.c:232
    #3 0x7e36a742a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #4 0x7e36a742a28a in __libc_start_main_impl ../csu/libc-start.c:360
    #5 0x5f323c840784 in _start (/home/shootduck/hotracer-experiments/0-day/hotracer/pilot/genmkvpwd/2026-08-01-1.9.0-jumbo-1+bleeding-9a336d8/john/build-asan/run/genmkvpwd+0x4784) (BuildId: 4875d0d0571be6b5aca6c0dbdb99f1729b7a84ad)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/shootduck/hotracer-experiments/0-day/hotracer/pilot/genmkvpwd/2026-08-01-1.9.0-jumbo-1+bleeding-9a336d8/john/build-asan/src/mkvlib.c:44 in nb_parts
Shadow bytes around the buggy address:
  0x528000003680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x528000003700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x528000003780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x528000003800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x528000003880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x528000003900: fa[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x528000003980: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x528000003a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x528000003a80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x528000003b00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x528000003b80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==785159==ABORTING

Potential fix

Apply the existing Markov limits before selecting an allocation branch. This is consistent with src/mkv.c, which already constrains the same parameters. It preserves the program's existing behavior of clamping oversized values while ensuring that all three allocation sites receive safe dimensions.

I applied this diff to a fresh checkout of the vulnerable commit, rebuilt it with the ASan command above, and reran the 16,777,216 input and both negative max_len invocation forms. All completed without an AddressSanitizer diagnostic.

diff --git a/src/genmkvpwd.c b/src/genmkvpwd.c
index de62d03..2d19662 100644
--- a/src/genmkvpwd.c
+++ b/src/genmkvpwd.c
@@ -224,6 +224,16 @@ int main(int argc, char * * argv)
 		end = atoll(argv[5]);
 
 	init_probatables(argv[1]);
+	if (max_lvl > MAX_MKV_LVL) {
+		fprintf(stderr, "Warning: Level = %u is too large (max = %d)\n",
+		        max_lvl, MAX_MKV_LVL);
+		max_lvl = MAX_MKV_LVL;
+	}
+	if (max_len > MAX_MKV_LEN) {
+		fprintf(stderr, "Warning: Maxlen = %u is too large (max = %d)\n",
+		        max_len, MAX_MKV_LEN);
+		max_len = MAX_MKV_LEN;
+	}
 
 	if (max_len == 0)
 	{
@@ -266,11 +276,6 @@ int main(int argc, char * * argv)
 		}
 		goto fin;
 	}
-	if (max_lvl>MAX_MKV_LVL) {
-		fprintf(stderr, "Warning: Level = %d is too large (max = %d)\n", max_lvl, MAX_MKV_LVL);
-		max_lvl = MAX_MKV_LVL;
-	}
-
 	nbparts = mem_alloc(256*(max_lvl+1)*sizeof(long long)*(max_len+1));
 	fprintf(stderr, "allocated %lu KB for nbparts\n", (unsigned long)(256UL*(max_lvl+1)*(max_len+1)*sizeof(long long)/1024));
 	memset(nbparts, 0, 256*(max_lvl+1)*(max_len+1)*sizeof(long long));

Artifacts

The PoC and the sanitizer report are packaged into this ZIP file.

artifacts.zip

[solardiz]

Not really a security issue, but thanks for reporting. Overlap with #5799 and/or #5798? Does #6036 fix this?