Repository navigation
Fixes #34933: Match FK referred columns to the parent table's column case - #34934
Conversation
…n case MySQL can keep a foreign key's referenced column as written in the FK definition (`REFERENCES limitadjustmentreasons (id)`) while the parent column is `Id`. Ingestion built referredColumns from that raw name and the server validates referred columns case-sensitively, so it rejected the whole child table with `Invalid column name id`, or silently dropped the FK when it was deferred to the end of the run. Resolve each referred column against the parent table already fetched from OpenMetadata: exact match first, then a single case-insensitive match. Use the resolved names for the relationship type too, which reported MANY_TO_MANY instead of MANY_TO_ONE. A referred column that still doesn't resolve is retried after all tables are ingested, then skipped with a warning instead of failing the table.
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
github-actions
Bot
added
Ingestion
safe to test
Code Review ✅ Approved🟡 Medium risk · Resolving and deferring foreign-key references changes ingestion metadata; unresolved references may be skipped, leaving relationship metadata incomplete. Fixes MySQL foreign key ingestion when the referred column name differs in case from the parent table's stored column. The fix resolves column names case-insensitively during constraint preparation, retries unresolved FKs after all tables are ingested, and corrects the relationship type from OptionsDisplay: compact → Counting what did not apply, without listing it. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
|
Describe your changes:
Fixes #34933
Ingestion rejected a whole MySQL table with
Invalid column name id(400) when one of its foreign keys referenced the parent column in a different case:REFERENCES parent (id)while the column isId. When the FK was deferred (e.g. on a first run), it was silently dropped instead. Full analysis and reproduction are in #34933.Root cause
foreign_key_checks=0before the parent table existed, or on MySQL 8.0.1–8.0.13. SQLAlchemy returns the name unchanged on 8.0.14+._prepare_foreign_constraintsalready loads the parent table from OpenMetadata, but it builtreferredColumnsfrom that raw name. The server checks referred columns case-sensitively, so it rejected the table, or rejected the end-of-run FK PATCH when the FK had been deferred.relationshipTypewas also wrong:MANY_TO_MANYinstead ofMANY_TO_ONE, because the parent's primary key wasn't found under the other casing.Changes
metadata/utils/constraints.py: newresolve_column_namesmaps column names from the source to the stored ones. An exact match wins, then a single case-insensitive match. It returnsNoneif a name matches no column or several._prepare_foreign_constraintsbuildsreferredColumnsand the relationship type from the resolved names.The fix is in ingestion rather than the server because the server's case-sensitive check is correct for case-sensitive sources, where a table can have both
idandID. Ingestion already normalizes a table's own constraint columns the same way innormalize_table_constraints.Needs a backport to 2.0.x (reported on 2.0.3). On
upstream/2.0this function differs only in lines next to the change.Type of change:
High-level design:
N/A: small bug fix (2 source files plus tests).
Tests:
Use cases covered
MANY_TO_ONE.idandIDexist, the exact match wins.Unit tests
ingestion/tests/unit/topology/database/test_common_db_source.py, classTestPrepareForeignConstraintsReferredColumnCase(8 cases). 7 fail onmain; the exact-match case passes on both, by design.deflines, which run before coverage starts).tests/unit/topology/databaseandtests/unit/utils: the only difference in results betweenmainand this branch is the new tests.Backend integration tests
Ingestion integration tests
ingestion/tests/integration/mysql/test_foreign_key_case.py. It creates an FK in the MySQL testcontainer that MySQL stores asREFERENCES fk_case_parent (id)against parent columnId(checked viaSHOW CREATE TABLE), runs the metadata workflow twice, and checks the stored FK. It fails onmainwithInvalid column name idand passes on this branch.Playwright (UI) tests
Manual testing performed
mysql:8.0(8.0.46), recreated the reported table, creating the child before the parent withSET foreign_key_checks = 0.SHOW CREATE TABLEshowsREFERENCES <parent> (id)while the parent column isId. MySQL enforces the FK (ERROR 1452 on a missing parent row).main, ranmetadata ingesttwice against a local OpenMetadata server (2.0.0-SNAPSHOT):Invalid column name id, logged only as a warning. The workflow reported 100% success and the FK was missing.Failed to flush entities to bulk API … 'referredColumns': ['…<parent>.id'], 'relationshipType': 'MANY_TO_MANY' … message='Invalid column name id' status=400, and the workflow failed.…<parent>.IdwithMANY_TO_ONE.UI screen recording / screenshots:
Not applicable.
Checklist:
Fixes <issue-number>: <short explanation>Fixes #<issue-number>above.