MDEV-36990 : SIGFPE in get_max_range_rowid_filter_elems_for_table

[pranavktiwari]

fixes MDEV-36990

Problem:

ALTER TABLE allowing a column defined as BINARY(0) to be used as part of a PRIMARY KEY or UNIQUE KEY can lead to a zero-length index column. This results in tab->file->ref_length becoming 0 inside the storage engine, which later causes a division-by-zero crash in get_max_range_rowid_filter_elems_for_table() during optimizer cost estimation for range queries.

Cause:

During key initialization in init_key_part_spec(), validation relied on column->flags & NOT_NULL_FLAG to detect non-nullable key columns. However, in ALTER TABLE ... CHANGE COLUMN, the Create_field is only partially initialized and does not yet propagate implicit constraints such as PRIMARY KEY ⇒ NOT NULL. As a result, NOT_NULL_FLAG may be 0 even for primary key columns, allowing key_part_length == 0 cases to bypass validation.

This leads to invalid index definitions where a primary key column contributes zero bytes to the index.

Fix:

Strengthen key validation in init_key_part_spec() by rejecting zero-length
key parts for columns that are effectively non-nullable in key context,
including PRIMARY KEY columns whose implicit NOT NULL property may not yet
be reflected in Create_field::flags during ALTER TABLE processing.

This prevents creation of invalid zero-length index definitions and avoids
propagation of ref_length=0 metadata that can later trigger optimizer
crashes.

[gemini-code-assist]

Code Review

This pull request modifies the key part specification initialization in sql/sql_table.cc to include primary keys in the condition check when key_part_length is zero. The feedback suggests adding explicit parentheses around the bitwise AND operation to clarify operator precedence and prevent potential compiler warnings when combined with the logical OR operator.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[sanja-byelkin]

I think that bug is in absence of NOT_NULL_FLAG for the primary index. Please investigate where it set and why have not set here.

Also pay attention to small things like spaces and correct markers.

[sanja-byelkin]

# is missing

[pranavktiwari]

done

[sanja-byelkin]

space is missing before ||

[pranavktiwari]

done

[pranavktiwari]

I think that bug is in absence of NOT_NULL_FLAG for the primary index. Please investigate where it set and why have not set here.

@sanja-byelkin
It will be null, because alter_info->create_list is coming out of parsed object and in the query we do not specify if this not nullable property hence this flag is not set.

ALTER TABLE t CHANGE COLUMN a a BINARY (0);

Here column a is nullable, though later it will be merged with the actual database filed which happens after this method.

[sanja-byelkin]

address also previous review changes