`ImageDataGenerator.flow_from_directory` models neither its iteration protocol nor the `(x, y)` batch tuple it yields

[khatchad]

What The Summary Models

The summary for tensorflow/keras/preprocessing/image/flow_from_directory returns a fresh instance of its own class and nothing else:

<class name="flow_from_directory" allocatable="true">
  <method name="do" descriptor="()LRoot;" numArgs="16" paramNames="self directory target_size ...">
    <new def="x" class="Ltensorflow/keras/preprocessing/image/flow_from_directory"/>
    <return value="x"/>
  </method>
</class>

The class declares no __iter__ and no __next__, and no summary anywhere creates the value the iterator actually produces.

What The API Does

A DirectoryIterator is an iterable of batches, and each batch is a two-element tuple (x, y): a rank-4 float32 image batch and a label array whose shape follows class_mode. The ordinary consuming idiom below therefore yields a tuple of two tensors at run time, where the analysis sees an opaque flow_from_directory instance with no element structure and no tensor typing on either position.

train_generator = ImageDataGenerator(rescale=1.0 / 255).flow_from_directory(directory, target_size=(112, 112), batch_size=512, class_mode="categorical")
train_dataset = iter(train_generator)
batch = next(train_dataset)     # (images, labels)

Why It Matters

This is the standard image-classification input pipeline, and the batch is normally forwarded straight into a training step that destructures it:

def train_step(inputs):
    images, labels = inputs
    ...

distributed_train_step(next(train_dataset))

Anything reasoning about that parameter's structure therefore has nothing to reason from: the tuple arity, the image batch's rank-4 float32 typing, and the label position all go missing at the generator boundary rather than at the destructuring. I hit this while reproducing ponder-lab/Hybridize-Functions-Refactoring#888, where the same tuple-parameter shape works when the tuple is visible in source and stays invisible when it arrives through this generator.

Suggested Shape

Give the class an iteration model whose element is a two-position container holding a dense rank-4 float32 tensor and a label tensor, so next(iter(gen)) carries the tuple structure. Both target_size and batch_size are summary parameters, so the image batch's extents are recoverable rather than ⊤ where the call passes literals. The label position's shape depends on class_mode, which is also a parameter, though a dtype-only element there would already be a large improvement over an opaque object.

Verified by reading the summary in com.ibm.wala.cast.python.ml/data/tensorflow.xml at the currently consumed release. I have not attributed any particular downstream verdict to it beyond the missing structure.

[khatchad]

The diagnosis and the shape of the fix look right to me, and the gap is not hypothetical: TensorFlow2.0-Examples reaches it in two files, 7-Utils/multi_gpu_train.py:44 and 5-Image_Segmentation/Unet/train.py:38, both with literal target_size and batch_size in the call. Three things to add before anyone implements it.

There Is A Direct Precedent, One Week Old

Issue #826 modeled enumerate as yielding (index, element) tuples. That is the same problem shape as this one: an iterable whose element is a heterogeneous two-position tuple, one position tensor-typed and one not. Whatever it established about giving an iteration model a structured element is the template here, and reusing it is preferable to inventing a second mechanism.

Its companion is worth knowing about too. Issue #409 added the drops pin because the pointer analysis aliases enumerate's integer index slot with the iterable's element type. A label position is not an integer index, but it is the second slot of a tuple whose first slot is a rank-4 image batch, so the same aliasing question should be asked rather than assumed away.

The Batch Axis Must Not Be The batch_size Literal

This is the part of the suggested shape I would change. A DirectoryIterator does not drop its remainder: when the image count is not a multiple of batch_size, the final batch of each epoch is short. Emitting the literal as a concrete extent therefore describes a batch that the loader does not always produce.

That matters more than a lost bit of precision, because a downstream consumer can turn a concrete extent into a hard failure. A signature naming 512 on the batch axis raises as soon as the short batch arrives, whereas a wildcard there accepts both. A concrete-looking extent that is wrong on the last iteration is worse than an unresolved one.

The axis is feed dependent in the sense the Dimension documentation uses, so Dynamic reads as the right classification, but that call should be made against the stated criterion rather than by analogy with mine. Note that the value here is produced by a Python generator yielding NumPy arrays rather than by a tf.data.Dataset, so there is no TensorShape to consult, which is exactly the case where the Dynamic versus Unresolved boundary deserves an explicit reading.

The non-batch image axes are a different matter. Those come from target_size and really are fixed by the resize, so they are recoverable as the issue says.

Two Smaller Gaps In The Recoverable-Parameter List

The rank-4 shape is (batch, height, width, channels), and channels comes from color_mode rather than from target_size. It is also a summary parameter and defaults to rgb, so it is recoverable, but it needs naming or the fourth axis has no source.

The label position under class_mode="categorical" is (batch, num_classes), and num_classes is discovered by counting subdirectories at call time. No summary parameter carries it. A dtype-only element there, as the issue already suggests as a fallback, looks like the right answer for that position rather than a second-best one.

[khatchad]

Following up with the downstream verdict the issue deliberately did not attribute, since it turns out to be concrete and it raises the priority.

What Is Emitted Today

In TensorFlow2.0-Examples, 7-Utils/multi_gpu_train.py, two functions receive a signature at 0.52.83:

@tf.function(input_signature=[tf.TensorSpec(shape=None, dtype=tf.float32)])
def train_step(inputs):
    images, labels = inputs

@tf.function(input_signature=[tf.TensorSpec(shape=None, dtype=tf.float32)])
def distributed_train_step(dataset_inputs):

The parameter of train_step is destructured into two on its first line, and its value comes from the generator this issue is about. A single TensorSpec is therefore wrong in kind rather than merely loose: it describes one tensor where the caller supplies a pair. This is the same defect as ponder-lab/Hybridize-Functions-Refactoring#888, arriving through the generator boundary instead of through a source-visible tuple, which is what the issue predicted would happen and is now observable.

Why That Makes This More Urgent Than A Precision Gap

A signature that is wrong in kind does not degrade a result, it stops the program. That is the same class as the signatures that had to be withheld in #884, #885 and #886, and it is the reason those were treated as correctness work rather than precision work.

And It Cannot Be Caught By Running Anything

The program reads its images from ./mnist/train under its own directory, and that data is not present. It therefore cannot be executed at all, by anyone, in the state the repository ships. Whatever the emitted signature does at trace time is unobservable.

That is the part that changes the calculus. For the functions in #884 through #886 there was a runtime failure to point at, which is what made them obviously urgent. Here there is not, and there never will be, so the only thing that can establish the specification is wrong is the static reasoning in this issue. An input pipeline that no runtime check can reach is exactly where a modeling gap survives indefinitely.

The sibling use in 5-Image_Segmentation/Unet/train.py emits nothing today, so multi_gpu_train.py is the whole of the observable surface for now.

Unchanged

Everything in my earlier comment still stands, in particular that the batch axis must not be emitted as the batch_size literal. If anything this reinforces it: a tuple element carrying a concrete batch extent would be wrong twice over.