diff --git a/doc/api/model/static-methods.md b/doc/api/model/static-methods.md index a0244b72f..64bb48f23 100644 --- a/doc/api/model/static-methods.md +++ b/doc/api/model/static-methods.md @@ -73,8 +73,8 @@ console.log(sylvester.fullName()); // --> 'Sylvester Stallone'. // Batch insert. On MySQL, this only works for models that -// have their identifiers already, as MySQL doesn't return -// the identifiers of _all_ inserted rows. +// have their identifiers already, or have no `idColumn`, as +// MySQL doesn't return the identifiers of _all_ inserted rows. const inserted = await Person.query().insert([ { firstName: 'Arnold', lastName: 'Schwarzenegger' }, { firstName: 'Sylvester', lastName: 'Stallone' } diff --git a/doc/api/query-builder/mutate-methods.md b/doc/api/query-builder/mutate-methods.md index 952fdac97..bc87dd0fe 100644 --- a/doc/api/query-builder/mutate-methods.md +++ b/doc/api/query-builder/mutate-methods.md @@ -18,11 +18,17 @@ properties - see [this recipe](/recipes/returning-tricks.html) for some examples `returning(['only', 'some', 'props'])` note that the result object will still contain the input properies **plus** the properties listed in `returning`. On other databases you can use the [insertAndFetch](/api/query-builder/mutate-methods.html#insertandfetch) method. -Batch inserts work on Postgres, SQL Server and SQLite (3.35 or newer), which return -the identifiers of _all_ inserted rows. On MySQL, they only work if all models have -their identifiers already, as MySQL only returns the id of the first inserted row. -If you need batch inserts of new rows on MySQL, you can use knex directly through -[knexQuery](/api/model/static-methods.html#static-knexquery). +Batch inserts work on Postgres, SQL Server and SQLite, which return the identifiers +of _all_ inserted rows. On MySQL, they only work if all models have their identifiers +already or the model has no `idColumn`, as MySQL only returns the id of the first +inserted row. If you need batch inserts of new rows on MySQL, you can use knex +directly through [knexQuery](/api/model/static-methods.html#static-knexquery). + +On SQLite, batch inserts of new rows need SQLite 3.35 or newer for `returning`, and +fail with a syntax error near "returning" on older versions. Columns that some rows +of a batch leave out get NULL instead of their defaults with knex's +`useNullAsDefault` option, and make knex throw without it. Batches of more than 500 +rows or 32766 values fail. `insert` writes whole values, so it rejects [FieldExpressions](/api/types/#type-fieldexpression) like `'meta:a.b'` that write parts of a json column with a [ValidationError](/api/types/#class-validationerror). Use [patch](/api/query-builder/mutate-methods.html#patch) for them. Keys with a `:` that the schema declares in `properties` are normal properties and columns, not field expressions. @@ -630,10 +636,11 @@ the case of a many-to-many relation, creates a join row to the join table. In the case of a belongs-to-many relation, adds the keys to the arrays of the owners. -Multiple items can be related by giving an array of identifiers: for has-many -and belongs-to-many relations on all databases, for many-to-many relations only -on Postgres and SQL Server, as it needs a batch insert. Belongs-to-one relations -take one identifier. +Multiple items can be related by giving an array of identifiers: for has-many, +belongs-to-many and many-to-many relations on all databases. On MySQL, the join +rows of many-to-many relations need to have their ids, which they have unless a +`through.modelClass` is used whose `idColumn` isn't made of the join columns. Belongs-to-one relations take +one identifier. The return value of the query is the number of affected items. On all databases except MySQL you can chain [returning('\*')](/api/query-builder/find-methods.html#returning) @@ -723,7 +730,7 @@ const numRelatedRows = await person console.log('movie 50 is now related to person 123 through `movies` relation'); ``` -Relate multiple (only works with postgres) +Relate multiple ```js const numRelatedRows = await Person.relatedQuery('movies') diff --git a/doc/release-notes/changelog.md b/doc/release-notes/changelog.md index 213bcc5b6..be7e2a5d9 100644 --- a/doc/release-notes/changelog.md +++ b/doc/release-notes/changelog.md @@ -56,7 +56,7 @@ ### Features -- `insert()` of several models works on SQLite (3.35 or newer) through `returning`, and on all databases when all models have their ids, e.g. join rows. `insertGraph()` and `upsertGraph()` insert in batches there too +- `insert()` of several models works on SQLite (3.35 or newer, up to 500 rows) through `returning`, and on all databases when all models have their ids or the model has no `idColumn`, e.g. join rows - `BelongsToManyRelation`, a relation where the owner holds an array of the related models' keys, e.g. `posts.tagIds = [1, 4, 7]` for `post.tags`, in a JSON column on all databases or a native array column on PostgreSQL. It supports finding the related models with `$relatedQuery()`, `relatedQuery()`, `withGraphFetched()`, `withGraphJoined()` and `joinRelated()`. `insert()`, `delete()`, `relate()` and `unrelate()` through the relation rewrite the arrays, `insertGraph()` and `upsertGraph()` write them, and `$setRelated()` and `$appendRelated()` keep them in sync. See [relations](/guide/relations.md) [#2667](https://github.com/Vincit/objection.js/issues/2667) - `Model.jsonColumnFormat` says how the database returns JSON columns: `'auto'` (default) leaves them as they are on PostgreSQL drivers and parses JSON text on others, `'text'` parses them strictly and throws for invalid JSON, `'parsed'` never parses them [#205](https://github.com/ditojs/objection/issues/205) - JSON field expressions work on MySQL and SQLite: in `ref()`, e.g. `ref('column:a.b').castInt()` in `select()`, `where()`, `orderBy()`, joins and relation mappings, as keys in `patch()` and `merge()`, e.g. `patch({ 'column:a.b': 1 })`, and in the `whereJson*()` methods, except for `whereJsonSupersetOf()` and `whereJsonSubsetOf()` on SQLite. objection writes the SQL of each database, `JSON_EXTRACT()` and `JSON_SET()` on MySQL and `json_extract()` and `json_set()` on SQLite, and casts to their types. `ref().castJson()` works on both too. See [JSON queries](/recipes/json-queries.md) [#276](https://github.com/Vincit/objection.js/issues/276) @@ -68,6 +68,7 @@ - The error of `upsertGraph()` queries with other query builder calls names `$query()`, which adds such a call too [#1168](https://github.com/Vincit/objection.js/issues/1168) - `upsertGraphAndFetch()` loads the relations of a `withGraphFetched()` on its query with their modifiers, named modifiers, aliases and options into the fetched models, like `insertGraphAndFetch()`, instead of ignoring the modifiers and aliases [#839](https://github.com/Vincit/objection.js/issues/839) +- On SQLite, `insert()` stores arrays in columns that aren't `jsonAttributes` as JSON, like objects, instead of `'[object Object]'` - A many-to-many `patch()` / `update()` with join table extras and `limit()` on MySQL and SQL Server can order by columns of the join table [#161](https://github.com/ditojs/objection/issues/161) - `cancelQuery()` with a single row in the `beforeInsert` hook of an `insert()` merges it into the inserted model, like an array of rows, instead of ignoring it [#66](https://github.com/ditojs/objection/issues/66) diff --git a/lib/queryBuilder/operations/InsertOperation.js b/lib/queryBuilder/operations/InsertOperation.js index abb39bbf1..a1e4362f5 100644 --- a/lib/queryBuilder/operations/InsertOperation.js +++ b/lib/queryBuilder/operations/InsertOperation.js @@ -27,20 +27,23 @@ export class InsertOperation extends QueryBuilderOperation { } async onBefore2(builder, result) { + await callBeforeInsert(builder, this.models); + // Without ids to return, rows can be inserted in batches on all databases. + // The hooks may give the models their ids, or cancel the query. if ( this.models.length > 1 && !hasBatchInsert(builder.knex()) && - !this.models.every((model) => model.$hasId()) + !builder.isExplicitlyResolvedOrRejected() && + this._hasIdsToReturn(builder) ) { throw new Error( - 'batch insert only works with Postgresql, SQL Server and SQLite, or with models that have their ids', + 'batch insert only works with Postgresql, SQL Server and SQLite, or with models that have their ids or no idColumn', ); - } else { - await callBeforeInsert(builder, this.models); - this.models.forEach(removeOverriddenValues); - return result; } + + this.models.forEach(removeOverriddenValues); + return result; } onBuildKnex(knexBuilder, builder) { @@ -55,7 +58,10 @@ export class InsertOperation extends QueryBuilderOperation { // conflict columns are returned too, so that the returned rows can // be matched to the models when some of the rows are ignored. knexBuilder = knexBuilder.returning(defaultReturning); - } else if (isSqlite(knex) && (builder.has('onConflict') || this.models.length > 1)) { + } else if ( + isSqlite(knex) && + (builder.has('onConflict') || (this.models.length > 1 && this._hasIdsToReturn(builder))) + ) { // Without a `returning` clause, SQLite returns the id of the last // inserted row, which belongs to a previous insert if the row was // ignored, and only one id for batch inserts. Use `returning` to find @@ -65,9 +71,9 @@ export class InsertOperation extends QueryBuilderOperation { } const modelClass = builder.modelClass(); - // knex inserts several rows into SQLite with `select ... union all`, which - // doesn't serialize objects and arrays like single rows do. - const shouldSerialize = isSqlite(knex) && this.models.length > 1; + // knex leaves arrays unserialized in SQLite inserts, and also objects when + // it inserts several rows with `select ... union all`. + const shouldSerialize = isSqlite(knex); return knexBuilder.insert( this.models.map((model) => { @@ -78,6 +84,13 @@ export class InsertOperation extends QueryBuilderOperation { ); } + _hasIdsToReturn(builder) { + return ( + builder.modelClass().getIdColumnArray().length > 0 && + !this.models.every((model) => model.$hasId()) + ); + } + onAfter1(builder, ret) { if (!Array.isArray(ret) || !ret.length || ret === this.models) { // Early exit if there is nothing to do. diff --git a/tests/integration/insert.js b/tests/integration/insert.js index dd8b0984d..3d073c1a0 100644 --- a/tests/integration/insert.js +++ b/tests/integration/insert.js @@ -209,7 +209,8 @@ export default (session) => { }); }); - if (session.isPostgres()) { + // MySQL doesn't return the ids of several inserted rows. + if (!session.isMySql()) { it('should accept an array', () => { let models = [ Model1.fromJson({ model1Prop1: 'hello 3' }), @@ -282,6 +283,42 @@ export default (session) => { }); }); + it('should accept a json array of models that get their ids in $beforeInsert', async () => { + let nextId = 200; + + class IdModel extends Model1 { + $beforeInsert(context) { + super.$beforeInsert(context); + this.id = nextId++; + } + } + + const inserted = await IdModel.query().insert([ + { model1Prop1: 'hello 200' }, + { model1Prop1: 'hello 201' }, + ]); + + expect(inserted.map((it) => it.id)).toEqual([200, 201]); + expect(await session.knex(Model1.getTableName()).whereIn('id', [200, 201])).toHaveLength(2); + }); + + it('should accept a json array of models whose insert a hook cancels', async () => { + class CancelModel extends Model1 { + static beforeInsert({ cancelQuery }) { + cancelQuery([]); + } + } + + await CancelModel.query().insert([ + { model1Prop1: 'cancelled 1' }, + { model1Prop1: 'cancelled 2' }, + ]); + + expect( + await session.knex(Model1.getTableName()).where('model1Prop1', 'like', 'cancelled%'), + ).toEqual([]); + }); + it('should accept a json array of models with ids', async () => { const inserted = await Model1.query().insert([ { id: 100, model1Prop1: 'hello 100' }, @@ -906,7 +943,8 @@ export default (session) => { }); }); - if (session.isPostgres()) { + // MySQL doesn't return the ids of several inserted rows. + if (!session.isMySql()) { it('should accept an array', () => { let inserted = null; @@ -1134,7 +1172,8 @@ export default (session) => { }); }); - if (session.isPostgres()) { + // MySQL doesn't return the ids of several inserted rows. + if (!session.isMySql()) { it('should accept an array', () => { let inserted = null; @@ -1557,7 +1596,8 @@ export default (session) => { }); }); - if (session.isPostgres()) { + // MySQL doesn't return the ids of several inserted rows. + if (!session.isMySql()) { it('should accept an array', () => { return Model1.relatedQuery('model1Relation2') .for(1) diff --git a/tests/integration/misc/issue205.js b/tests/integration/misc/issue205.js index 425cd455a..8d82cf622 100644 --- a/tests/integration/misc/issue205.js +++ b/tests/integration/misc/issue205.js @@ -60,7 +60,7 @@ export default (session) => { return knex('issue205Item').delete(); }); - // Not all databases support batch inserts: + // MySQL doesn't return the ids of several inserted rows: const insertItems = async (items) => { for (const item of items) { await Item.query().insert(item); diff --git a/tests/integration/misc/issue2305.js b/tests/integration/misc/issue2305.js index f0fe3d9dd..e28ce05d9 100644 --- a/tests/integration/misc/issue2305.js +++ b/tests/integration/misc/issue2305.js @@ -70,8 +70,7 @@ export default (session) => { expect(await rows()).toEqual([{ name: 'a', runs: 1 }]); }); - // Only Postgres supports batch inserts. - it.skipIf(!session.isPostgres())('should insert multiple models', async () => { + it('should insert multiple models', async () => { const jobs = await JobExecution.query().insert([ { name: 'a', runs: 1 }, { name: 'b', runs: 2 }, diff --git a/tests/integration/misc/sqliteBatchInsert.js b/tests/integration/misc/sqliteBatchInsert.js new file mode 100644 index 000000000..8e1f9d8d9 --- /dev/null +++ b/tests/integration/misc/sqliteBatchInsert.js @@ -0,0 +1,101 @@ +import { describe, it, expect, beforeAll, beforeEach, afterAll } from 'vitest'; +import { Model } from 'objection'; + +export default (session) => { + describe.skipIf(!session.isSqlite())('insert() of several models on SQLite', () => { + const { knex } = session; + + class Item extends Model { + static get tableName() { + return 'batch_insert_item'; + } + } + + class Pair extends Model { + static get tableName() { + return 'batch_insert_pair'; + } + + static get idColumn() { + return ['a', 'b']; + } + } + + const dropTables = () => + knex.schema.dropTableIfExists('batch_insert_item').dropTableIfExists('batch_insert_pair'); + + beforeAll(async () => { + await dropTables(); + await knex.schema + .createTable('batch_insert_item', (table) => { + table.increments('id'); + table.string('name').unique(); + table.text('data'); + }) + .createTable('batch_insert_pair', (table) => { + table.integer('a'); + table.integer('b'); + table.string('name'); + table.primary(['a', 'b']); + }); + }); + + beforeEach(async () => { + await knex('batch_insert_item').delete(); + await knex('batch_insert_pair').delete(); + }); + + afterAll(dropTables); + + it('should store objects and arrays as JSON, in one row and in several', async () => { + await Item.query(knex).insert({ name: 'a', data: [1, 2] }); + await Item.query(knex).insert([ + { name: 'b', data: { b: 1 } }, + { name: 'c', data: [3] }, + ]); + + expect(await knex('batch_insert_item').select('name', 'data').orderBy('name')).toEqual([ + { name: 'a', data: '[1,2]' }, + { name: 'b', data: '{"b":1}' }, + { name: 'c', data: '[3]' }, + ]); + }); + + it('should return the ids of the inserted rows with onConflict().ignore()', async () => { + await knex('batch_insert_item').insert({ name: 'a' }); + + const [a, b] = await Item.query(knex) + .insert([{ name: 'a' }, { name: 'b' }]) + .onConflict('name') + .ignore(); + + expect(a.id).toBeUndefined(); + expect(b.id).toBe((await knex('batch_insert_item').where('name', 'b').first()).id); + }); + + it('should insert models with composite ids', async () => { + const pairs = await Pair.query(knex).insert([ + { a: 1, b: 2, name: 'x' }, + { a: 2, b: 1, name: 'y' }, + ]); + + expect(pairs.map((pair) => pair.$id())).toEqual([ + [1, 2], + [2, 1], + ]); + expect(await knex('batch_insert_pair').orderBy('a')).toHaveLength(2); + }); + + it('should not return anything for models with ids', () => { + const sql = Item.query(knex) + .insert([ + { id: 1, name: 'a' }, + { id: 2, name: 'b' }, + ]) + .toKnexQuery() + .toString(); + + expect(sql).not.toMatch(/returning/); + }); + }); +}; diff --git a/tests/integration/relate.js b/tests/integration/relate.js index dabec8780..f46b83dbf 100644 --- a/tests/integration/relate.js +++ b/tests/integration/relate.js @@ -345,29 +345,29 @@ export default (session) => { }); }); - if (session.isPostgres()) { - it('should relate (multiple values)', () => { - return Model2.query() - .where('id_col', 1) - .first() - .then((model) => { - return model.$relatedQuery('model2Relation1').relate([5, 6]); - }) - .then((res) => { - expect(res).toBe(2); - return session.knex('Model1Model2').orderBy('id'); - }) - .then((rows) => { - expect(rows).toHaveLength(6); - expect(rows.filter((it) => it.model2Id === 1 && it.model1Id === 3)).toHaveLength(1); - expect(rows.filter((it) => it.model2Id === 1 && it.model1Id === 5)).toHaveLength(1); - expect(rows.filter((it) => it.model2Id === 1 && it.model1Id === 6)).toHaveLength(1); - expect(rows.filter((it) => it.model2Id === 2 && it.model1Id === 4)).toHaveLength(1); - expect(rows.filter((it) => it.model2Id === 2 && it.model1Id === 5)).toHaveLength(1); - expect(rows.filter((it) => it.model2Id === 2 && it.model1Id === 6)).toHaveLength(1); - }); - }); + it('should relate (multiple values)', () => { + return Model2.query() + .where('id_col', 1) + .first() + .then((model) => { + return model.$relatedQuery('model2Relation1').relate([5, 6]); + }) + .then((res) => { + expect(res).toBe(2); + return session.knex('Model1Model2').orderBy('id'); + }) + .then((rows) => { + expect(rows).toHaveLength(6); + expect(rows.filter((it) => it.model2Id === 1 && it.model1Id === 3)).toHaveLength(1); + expect(rows.filter((it) => it.model2Id === 1 && it.model1Id === 5)).toHaveLength(1); + expect(rows.filter((it) => it.model2Id === 1 && it.model1Id === 6)).toHaveLength(1); + expect(rows.filter((it) => it.model2Id === 2 && it.model1Id === 4)).toHaveLength(1); + expect(rows.filter((it) => it.model2Id === 2 && it.model1Id === 5)).toHaveLength(1); + expect(rows.filter((it) => it.model2Id === 2 && it.model1Id === 6)).toHaveLength(1); + }); + }); + if (session.isPostgres()) { it('should relate with onConflict().ignore()', async () => { const sql = await Model2.relatedQuery('model2Relation1') .for(1) @@ -381,8 +381,12 @@ export default (session) => { 'insert into "Model1Model2" ("model1Id", "model2Id") values (?, ?) on conflict ("model1Id") do nothing returning "model1Id"', ); }); + } - it('should return the inserted join rows with returning()', async () => { + // MySQL has no `returning()`. + it.skipIf(session.isMySql())( + 'should return the inserted join rows with returning()', + async () => { const model = await Model2.query().findById(1); const result = await model .$relatedQuery('model2Relation1') @@ -400,8 +404,8 @@ export default (session) => { const rows = await session.knex('Model1Model2').orderBy('id'); expect(rows).toHaveLength(6); - }); - } + }, + ); it('should relate (object value)', () => { return Model2.query()