Fix select narrowing of snake case models, distinctOn(), clearSelect() and jsonExtract() (#16)

doc/guide/models.md

@@ -345,7 +345,7 @@ people[0].petCount; // number | string
 people[0].lastName; // Error: not selected
 ```
 
-`select('*')` and `select('persons.*')` keep all columns, `first()`, `findById()`, `page()` and `withGraphFetched()` keep the narrowed rows, and repeated selections add up.
+`select('*')` and `select('persons.*')` keep all columns, `first()`, `findById()`, `page()` and `withGraphFetched()` keep the narrowed rows, and repeated selections add up. `clearSelect()` gives back all columns. `distinctOn()` doesn't select its columns and `jsonExtract()` isn't a selection: they don't narrow the rows, and `jsonExtract()` adds its alias to them, of unknown type.
 
 Models that map their columns with `snakeCaseMappers()` declare it, see [Snake case to camel case conversion](/recipes/snake-case-to-camel-case-conversion.html). Queries for models without declared columns, like the base `Model`, and queries in generic code, e.g. in the methods of custom query builders, aren't checked. After `withGraphFetched()` on a generic query builder, e.g. `this.$query()` in a model method, TypeScript resolves the narrowed query builder to a union, on which only `ref()` and `raw()` columns are accepted.
 

doc/release-notes/changelog.md

@@ -32,7 +32,8 @@
 - Custom query builders declare one `RebindType` member instead of the five `ArrayQueryBuilderType`, `SingleQueryBuilderType`, `MaybeSingleQueryBuilderType`, `NumberQueryBuilderType` and `PageQueryBuilderType` members, which are removed: `declare RebindType: MyQueryBuilder<this['~M'], this['~R']>`. The helper types `SingleQueryBuilder<QB>` etc. take a query builder, and `WithGraphModelQueryBuilder` and `ReturningMethod` are removed. See [migration](/release-notes/migration.md#typescript-custom-query-builders-declare-rebindtype) [#125](https://github.com/ditojs/objection/issues/125) [#319](https://github.com/Vincit/objection.js/issues/319)
 - `static jsonSchema` is typed `ReadonlyJSONSchema`, a JSON schema with readonly arrays, so that it takes schemas declared `as const`. A model's schema read through the static type can't be passed where a `JSONSchema` is expected, unless the model declares `static jsonSchema: JSONSchema`. See [migration](/release-notes/migration.md#typescript-model-jsonschema-is-a-readonly-schema) [#16](https://github.com/ditojs/objection/issues/16)
 - `insert()`, `patch()`, `update()`, their `*AndFetch*()` and `*ById()` variants, `merge()` and `relate()` reject relation properties, which they ignore at runtime: they take a `PartialModelRow<M>`, a model instance or its properties without relations, instead of `PartialModelObject<M>`. Values whose type includes the relation properties are rejected too, e.g. `Partial<M>`, `ModelObject<M>` or `PartialModelObject<M>` of models with relations. `insertGraph()` and `upsertGraph()` take relations. See [migration](/release-notes/migration.md#typescript-insert-patch-and-update-reject-relation-data) [#53](https://github.com/ditojs/objection/issues/53)
-- The typings check the column names and values of queries against the models, and the relation names of relation expressions. Columns qualified with relation paths like `pets:owner.name` are checked against the related models, in any order of the joins. Dynamic column names, columns of other queries and aliases of selections outside of `orderBy()` need `ref()`. `select()` and the aggregates narrow the result rows to the selected columns and aliases, whose aliases `orderBy()` takes too. Models that use `snakeCaseMappers()` declare `ColumnNameMapping: 'snakeCase'` (or `'upperSnakeCase'` with `upperCase: true`) to check the snake case names. Opt out with `TypeConfig { strict: false }`. See [migration](/release-notes/migration.md#typescript-queries-are-checked-against-the-models) [#16](https://github.com/ditojs/objection/issues/16)
+- The typings check the column names and values of queries against the models, and the relation names of relation expressions. Columns qualified with relation paths like `pets:owner.name` are checked against the related models, in any order of the joins. Dynamic column names, columns of other queries and aliases of selections outside of `orderBy()` need `ref()`. `select()` and the aggregates narrow the result rows to the selected columns and aliases, whose aliases `orderBy()` takes too, and `jsonExtract()` adds its alias to the rows. Models that use `snakeCaseMappers()` declare `ColumnNameMapping: 'snakeCase'` (or `'upperSnakeCase'` with `upperCase: true`) to check the snake case names. Opt out with `TypeConfig { strict: false }`. See [migration](/release-notes/migration.md#typescript-queries-are-checked-against-the-models) [#16](https://github.com/ditojs/objection/issues/16)
+- `distinctOn()` is typed to take column names, `raw()` and `ref()`, like `groupBy()`, instead of any selection, as it doesn't select its columns.
 - `ModelObject`, `PartialModelObject` and the other property types leave out properties starting with `$`, which objection treats as internal, like `toJSON()` does. See [migration](/release-notes/migration.md#typescript-queries-are-checked-against-the-models) [#16](https://github.com/ditojs/objection/issues/16)
 - `$relatedQuery()` of a to-one relation, e.g. `person.$relatedQuery('parent')`, resolves to `Model | undefined`, as it does at runtime when there is no related row. See [migration](/release-notes/migration.md#typescript-relatedquery-of-to-one-relations-may-resolve-to-undefined) [#2355](https://github.com/Vincit/objection.js/issues/2355)
 - `toJSON()` and `$toJson()` return `ModelJson<M>`: the properties without methods, and the related models as JSON too, instead of `ModelObject<M>` with model instances. See [migration](/release-notes/migration.md#typescript-tojson-returns-modeljson) [#2380](https://github.com/Vincit/objection.js/issues/2380)

tests/ts/strict/declarations.ts

@@ -32,3 +32,7 @@ export function graph() {
 export function custom() {
   return CustomPerson.query().select('id').someCustomMethod();
 }
+
+export function extraction() {
+  return Person.query().select('id').jsonExtract('address', '$.street', 'street');
+}

tests/ts/strict/select.ts

@@ -126,3 +126,84 @@ async function indexSignature() {
   const row: Record<string, unknown> = (await Person.query().select('id', 'firstName as name'))[0];
   const counts: Record<string, unknown> | undefined = await Person.query().count('id as n').first();
 }
+
+async function distinctOn() {
+  // distinctOn() checks its columns, but doesn't select them.
+  const all = await Person.query().distinctOn('id');
+  all[0].firstName;
+  const selected = await Person.query().distinctOn('id').select('firstName');
+  selected[0].firstName;
+  // @ts-expect-error not selected
+  selected[0].id;
+  // @ts-expect-error
+  await Person.query().distinctOn('nme');
+}
+
+async function clearSelect() {
+  // clearSelect() selects all columns again.
+  const cleared = await Person.query().select('id as key').clearSelect();
+  cleared[0].firstName;
+  // @ts-expect-error not selected
+  cleared[0].key;
+  const reselected = await Person.query().select('id').clearSelect().select('firstName');
+  reselected[0].firstName;
+  // @ts-expect-error not selected
+  reselected[0].id;
+  const first = await Person.query().count('id as n').clearSelect().first();
+  type _first = Expect<Equal<typeof first, Person | undefined>>;
+  const custom = CustomPerson.query().select('id').clearSelect().someCustomMethod();
+  (await custom)[0].firstName;
+}
+
+async function jsonExtract() {
+  // jsonExtract() adds its alias, of unknown type, to all columns or to the
+  // selected ones, before or after select().
+  const extracted = await Person.query().jsonExtract('address', '$.street', 'street');
+  type _street = Expect<Equal<(typeof extracted)[number]['street'], unknown>>;
+  extracted[0].firstName;
+  const before = await Person.query().jsonExtract('address', '$.street', 'street').select('id');
+  before[0].street;
+  // @ts-expect-error not selected
+  before[0].firstName;
+  const after = await Person.query().select('id').jsonExtract('address', '$.street', 'street');
+  after[0].street;
+  // @ts-expect-error not selected
+  after[0].firstName;
+  await Person.query().jsonExtract('address', '$.street', 'street').orderBy('street');
+  // @ts-expect-error
+  await Person.query().jsonExtract('address', '$.street', 'street').orderBy('town');
+  const row: Record<string, unknown> = after[0];
+  // It survives clearSelect(), as it isn't a select, and withGraph*().
+  const cleared = await Person.query().jsonExtract('address', '$.street', 'street').clearSelect();
+  cleared[0].street;
+  const clearedBefore = await Person.query()
+    .jsonExtract('address', '$.street', 'street')
+    .select('id')
+    .clearSelect();
+  clearedBefore[0].street;
+  clearedBefore[0].firstName;
+  const clearedAfter = await Person.query()
+    .select('id')
+    .jsonExtract('address', '$.street', 'street')
+    .clearSelect()
+    .first();
+  type _clearedAfter = Expect<Equal<NonNullable<typeof clearedAfter>['street'], unknown>>;
+  clearedAfter?.firstName;
+  const fetched = await Person.query()
+    .jsonExtract('address', '$.street', 'street')
+    .withGraphFetched('pets');
+  fetched[0].street;
+  fetched[0].pets.length;
+  const joined = await Person.query()
+    .jsonExtract('address', '$.street', 'street')
+    .withGraphJoined('pets')
+    .first()
+    .throwIfNotFound();
+  joined.street;
+  joined.pets.length;
+  // Extractions without aliases add any properties.
+  const unnamed = await Person.query()
+    .select('id')
+    .jsonExtract([['address', '$.street']]);
+  unnamed[0].street;
+}

tests/ts/strict/snake-case.ts

@@ -55,6 +55,8 @@ async function queries() {
   rows[0].mainUrl.toUpperCase();
   // @ts-expect-error
   rows[0].first_name;
+  // @ts-expect-error not selected
+  rows[0].createdAt;
   // orderBy() takes the aliases as written in SQL.
   await Person.query().select('home_url as main_url').orderBy('main_url');
   // @ts-expect-error
@@ -98,6 +100,8 @@ async function upperCase() {
   const rows = await UpperModel.query().select('FIRST_NAME', 'ID as RECORD_ID');
   rows[0].firstName.toUpperCase();
   rows[0].recordId.toFixed();
+  // @ts-expect-error not selected
+  rows[0].id;
   await UpperModel.query().select('ID as RECORD_ID').orderBy('RECORD_ID');
 }
 

typings/objection/index.d.ts

@@ -24,6 +24,8 @@ declare const relationKind: unique symbol;
 declare const generated: unique symbol;
 // Phantom brand of the rows narrowed by `select()`, holding their SQL aliases.
 declare const selected: unique symbol;
+// Phantom brand of the rows with `jsonExtract()` aliases.
+declare const extracted: unique symbol;
 
 export const raw: RawFunction;
 export const val: ValueFunction;
@@ -418,8 +420,9 @@ type WithGraphResult<R, T> = R extends readonly (infer I)[]
 /**
  * Converts the query builder NQB to the same result kind as QB:
  * array, page, single or maybe-single. Rows that `select()` narrowed in
- * strict mode get the relations of tree T. Any other result type, e.g. after
- * `castTo()` or `patch()`, falls back to QB, without narrowing.
+ * strict mode or that have `jsonExtract()` values get the relations of tree T
+ * instead. Any other result type, e.g. after `castTo()` or `patch()`, falls
+ * back to QB, without narrowing.
  * QB's model type M and result type R are separate parameters so that QB
  * only appears in the result, and TypeScript can relate the kinds of two
  * query builders, e.g. of a custom query builder and its base class, through
@@ -432,17 +435,19 @@ export type WithResultKind<
   M extends Model = ModelType<QB>,
   R = ResultType<QB>,
   T = {},
-> = [R] extends [M[]]
-  ? ArrayQueryBuilder<NQB>
-  : [R] extends [Page<M>]
-    ? PageQueryBuilder<NQB>
-    : [R] extends [object | undefined]
-      ? [R] extends [M]
-        ? SingleQueryBuilder<NQB>
-        : [R] extends [M | undefined]
-          ? MaybeSingleQueryBuilder<NQB>
-          : Rebind<QB, M, WithGraphResult<R, T>>
-      : QB;
+> = [R] extends [ExtractedRows]
+  ? Rebind<QB, M, WithGraphResult<R, T>>
+  : [R] extends [M[]]
+    ? ArrayQueryBuilder<NQB>
+    : [R] extends [Page<M>]
+      ? PageQueryBuilder<NQB>
+      : [R] extends [object | undefined]
+        ? [R] extends [M]
+          ? SingleQueryBuilder<NQB>
+          : [R] extends [M | undefined]
+            ? MaybeSingleQueryBuilder<NQB>
+            : Rebind<QB, M, WithGraphResult<R, T>>
+        : QB;
 
 /**
  * The item of the result R of a query for model M, which `first()`,
@@ -1027,25 +1032,55 @@ type StrictSelectionObject<M, O> = {
 /**
  * The query builder QB with its result rows narrowed to the properties P in
  * strict mode, next to the model's methods and relations, as after `select()`
- * and the aggregates, marked as Selected. Repeated selections add up. Queries
+ * and the aggregates, marked with SelectedBrand. Repeated selections add up. Queries
  * that aren't checked, e.g. in generic code, aren't narrowed.
  */
-type NarrowRows<QB extends AnyQueryBuilder, P, All extends boolean = false> = IfStrict<
+type NarrowRows<QB extends AnyQueryBuilder, P, All extends boolean = false> = RowsQueryBuilder<
+  QB,
+  SelectBase<ResultItem<ResultType<QB>, ModelType<QB>>, ModelType<QB>, All> & P & SelectedBrand<{}>
+>;
+
+/**
+ * The query builder QB with the item of its result replaced by Row in strict
+ * mode. Queries that aren't checked keep their rows.
+ */
+type RowsQueryBuilder<QB extends AnyQueryBuilder, Row> = IfStrict<
   QB,
   IsConcreteModel<ModelType<QB>> extends true
-    ? Rebind<
-        QB,
-        ModelType<QB>,
-        SelectResult<
-          ResultType<QB>,
-          SelectBase<ResultItem<ResultType<QB>, ModelType<QB>>, ModelType<QB>, All> &
-            P &
-            SelectedBrand<{}>
-        >
-      >
+    ? Rebind<QB, ModelType<QB>, SelectResult<ResultType<QB>, Row>>
     : QB
 >;
 
+/**
+ * The query builder QB after `clearSelect()`, which selects all columns again,
+ * next to the values of `jsonExtract()`.
+ */
+type ClearSelectQueryBuilder<QB extends AnyQueryBuilder> =
+  ResultItem<ResultType<QB>, ModelType<QB>> extends infer Item
+    ? Item extends SelectedBrand<unknown>
+      ? RowsQueryBuilder<
+          QB,
+          ModelType<QB> &
+            (Item extends ExtractedBrand<infer A>
+              ? ExtractedProperties<ModelType<QB>, keyof A & string>
+              : {})
+        >
+      : QB
+    : never;
+
+/**
+ * The query builder QB after `jsonExtract()` with the alias A, which adds the
+ * extracted value to the rows. It isn't a select, so it doesn't narrow them.
+ */
+type JsonExtractQueryBuilder<QB extends AnyQueryBuilder, A extends string> = RowsQueryBuilder<
+  QB,
+  ResultItem<ResultType<QB>, ModelType<QB>> & ExtractedProperties<ModelType<QB>, A>
+>;
+
+type ExtractedProperties<M, A extends string> = {
+  [K in ResultKey<M, A>]: unknown;
+} & ExtractedBrand<{ [K in A]: true }>;
+
 /**
  * The query builder QB after selecting C. Selections of all columns keep the
  * columns in the rows.
@@ -1108,6 +1143,17 @@ type AggregateType<M, Col extends string, V extends AggregateValue> = V extends
  */
 export type SelectedBrand<A> = { readonly [selected]: A };
 
+/**
+ * The rows with values of `jsonExtract()` carry their SQL aliases A apart from
+ * SelectedBrand, as they aren't narrowed.
+ */
+export type ExtractedBrand<A> = { readonly [extracted]: A };
+
+// The results whose rows have `jsonExtract()` values, which `withGraph*()`
+// keeps instead of converting the result kind:
+type ExtractedRows =
+  ExtractedBrand<unknown>[] | Page<ExtractedBrand<unknown>> | ExtractedBrand<unknown> | undefined;
+
 /**
  * The base of the rows of a selection: the model without its columns for the
  * first one, the rows selected so far for further ones, and the model's
@@ -1117,7 +1163,7 @@ type SelectBase<Item, M, All extends boolean> = All extends true
   ? M & Item
   : Item extends SelectedBrand<unknown>
     ? Item
-    : Omit<Item, ColumnName<M>>;
+    : Omit<Item, InsertablePropertyNames<M>>;
 
 // Selections that can't be resolved, e.g. raw ones, add any properties of
 // unknown types, under any aliases.
@@ -1187,8 +1233,9 @@ type CamelCase<S extends string> = S extends `${infer H}_${infer T}`
  * columns.
  */
 type OutputColumn<QB extends AnyQueryBuilder> =
-  ResultItem<ResultType<QB>, ModelType<QB>> extends SelectedBrand<infer A>
-    ? keyof A & string
+  ResultItem<ResultType<QB>, ModelType<QB>> extends infer Item
+    ? | (Item extends SelectedBrand<infer A> ? keyof A & string : never)
+      | (Item extends ExtractedBrand<infer A> ? keyof A & string : never)
     : never;
 
 /**
@@ -1279,17 +1326,26 @@ export interface JsonExtraction {
   singleValue?: boolean;
 }
 
+// Extractions without an alias, or with several, add any properties.
 export interface JsonExtract<out QB extends AnyQueryBuilder> {
   // These must come first so that we get autocomplete.
-  <QBP extends QB, const C extends string = never>(
+  <QBP extends QB, const C extends string = never, const A extends string = string>(
     column: ColumnArg<ModelType<QBP>, C>,
     path: string,
-    alias?: string,
+    alias?: A,
     singleValue?: boolean,
-  ): QB;
+  ): JsonExtractQueryBuilder<QBP, A>;
 
-  (column: RefArg<QB>, path: string, alias?: string, singleValue?: boolean): QB;
-  (column: JsonExtraction[] | any[][], singleValue?: boolean): QB;
+  <QBP extends QB, const A extends string = string>(
+    column: RefArg<QBP>,
+    path: string,
+    alias?: A,
+    singleValue?: boolean,
+  ): JsonExtractQueryBuilder<QBP, A>;
+  <QBP extends QB>(
+    column: JsonExtraction[] | any[][],
+    singleValue?: boolean,
+  ): JsonExtractQueryBuilder<QBP, string>;
 }
 
 export interface JsonSet<out QB extends AnyQueryBuilder> {
@@ -1734,6 +1790,10 @@ export interface HasMethod {
   (selector: string | RegExp): boolean;
 }
 
+export interface ClearSelectMethod<out QB extends AnyQueryBuilder> {
+  <QBP extends QB>(): ClearSelectQueryBuilder<QBP>;
+}
+
 export interface ClearMethod<out QB extends AnyQueryBuilder> {
   (selector: string | RegExp): QB;
 }
@@ -1962,7 +2022,8 @@ export class QueryBuilder<M extends Model, R = M[]> implements CatchablePromiseL
   columns: SelectMethod<this>;
   column: SelectMethod<this>;
   distinct: SelectMethod<this>;
-  distinctOn: SelectMethod<this>;
+  // distinctOn() takes columns like groupBy(), and doesn't select them.
+  distinctOn: GroupByMethod<this>;
   as: AsMethod<this>;
 
   from: FromMethod<this>;
@@ -2285,7 +2346,7 @@ export class QueryBuilder<M extends Model, R = M[]> implements CatchablePromiseL
   has: HasMethod;
   clear: ClearMethod<this>;
 
-  clearSelect: IdentityMethod<this>;
+  clearSelect: ClearSelectMethod<this>;
   clearOrder: IdentityMethod<this>;
   clearWhere: IdentityMethod<this>;
   clearWithGraph: IdentityMethod<this>;