Skip to content

Check that training data is likely normalized or standardized correctly #81

Description

@kwinkunks

If data was split before scaling then...

Normalized: the min and max (eg 0 and 1, or -1 and +1) should be present in the training data. (Not so for test or application data.)

Standardized: the training data should be standard normal, ie pass is_standard_normal() (in particular, should have mean close to 0 and stdev of 1).

See also #6

OTOH, if scale then split...

Normalized: probably can't tell unless min or max happens to be in test split.

Standardized: For the mean, I think can check if that gets closer to 0, as long as we know how many training samples there were. For stdev, then possible to tell only by putting all the data together? Seems expensive.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions