From 0ee450791d798c9a81271c232ece52fd4a432c9e Mon Sep 17 00:00:00 2001 From: labkey-matthewb Date: Fri, 10 Jul 2026 14:49:56 -0700 Subject: [PATCH] Add hints to many parse error messages. Update LabKeySql.md --- .../query/controllers/prompts/LabKeySql.md | 640 ++++++++---------- .../src/org/labkey/query/sql/QMethodCall.java | 4 +- query/src/org/labkey/query/sql/SqlBase.g | 14 +- query/src/org/labkey/query/sql/SqlParser.java | 148 +++- .../src/org/labkey/query/sql/SyntaxHints.java | 150 ++++ 5 files changed, 594 insertions(+), 362 deletions(-) create mode 100644 query/src/org/labkey/query/sql/SyntaxHints.java diff --git a/query/src/org/labkey/query/controllers/prompts/LabKeySql.md b/query/src/org/labkey/query/controllers/prompts/LabKeySql.md index 51666e4d111..432f6111e81 100644 --- a/query/src/org/labkey/query/controllers/prompts/LabKeySql.md +++ b/query/src/org/labkey/query/controllers/prompts/LabKeySql.md @@ -1,415 +1,367 @@ ### **LabKey SQL Documentation** -LabKey SQL is a unique SQL dialect that extends standard SQL functionality with features tailored for the LabKey Server platform, particularly for scientific data management. -LabKey SQL only implements data read operations. It does not support INSERT/UPDATE/DELETE, nor does it support creating or altering tables. +LabKey SQL is a SQL dialect that extends standard SQL with features tailored for the LabKey Server platform. Queries are checked against the user's permissions and cross-compiled to the underlying database (PostgreSQL, or MS SQL Server for some premium installations). + +Core facts: +* **SELECT only.** No INSERT/UPDATE/DELETE, no DDL. A statement is: `[PARAMETERS(...)] [WITH ...] SELECT ... [set ops] [ORDER BY ...] [LIMIT n]`. +* Keywords, function names, schema/table/column names are **case-insensitive**. +* Comments: `-- line` and `/* block */`. +* If a `validateSQL` tool is available, use it to check syntax before saving or executing a query. Note that parse-time errors are descriptive, but type errors and dialect issues may only surface at execution time as an unhelpful generic message — apply the rules below proactively. ----- -### **1. Lookups and Joins** - -LabKey SQL simplifies data joining by providing an intuitive **lookup syntax** that often eliminates the need for explicit `JOIN` statements. - -* **Syntax:** - `SELECT parent_table.lookup_column.target_column FROM parent_table` -* **Functionality:** - This syntax allows you to access columns from a foreign table by following the lookup relationship with dot notation. This is a powerful feature for simplifying queries that involve related data. -* **Examples:** - * **Simple Lookup:** To retrieve a participant's gender from the `Demographics` table using a lookup from the `PhysicalExam` table: - ```sql - SELECT PhysicalExam.ParticipantId, PhysicalExam.weight_kg, Demographics.Gender, Demographics.Height - FROM PhysicalExam - ``` - * **Joining Across Folders:** To join data from a `Demographics` table in one folder with a `Languages` list in a different folder: - ```sql - JOIN "/Other/Folder".lists.Languages ON Demographics.Language=Languages.Language - ``` +### **1. NOT SUPPORTED — Write This Instead** + +LabKey SQL rejects many constructs that are valid in PostgreSQL/ANSI SQL. **Check generated SQL against this table first.** + +| Do NOT write | Why | Write instead | +| --- | --- | --- | +| `ROW_NUMBER() OVER (...)`, `RANK`, `LAG`, `SUM(x) OVER (...)` | No window functions, no `OVER` | Self-join or correlated subquery | +| `LIMIT 10 OFFSET 20`, `FETCH FIRST n ROWS`, `TOP n` | Only `LIMIT ` exists; no OFFSET | `LIMIT 10`; page via client API `maxRows`/`offset` | +| `GROUP BY 1` | Ordinals/constants rejected in GROUP BY (error: "Expression in Group By clause must not be a constant") | Repeat the expression: `GROUP BY YEAR(Date)`. (`ORDER BY 1` **is** allowed.) | +| `EXTRACT(YEAR FROM d)` | Not supported | `YEAR(d)`, `MONTH(d)`, `DAYOFMONTH(d)`, `HOUR(d)`, … | +| `CURRENT_DATE`, `CURRENT_TIMESTAMP` | Resolve as (missing) columns | `CURDATE()`, `NOW()`, `CURTIME()` | +| `col::integer` | No `::` cast | `CAST(col AS INTEGER)` | +| `SUM(DISTINCT x)`, `AVG(DISTINCT x)` | `DISTINCT` only inside `COUNT()` and `GROUP_CONCAT()` | Aggregate over a `SELECT DISTINCT` subquery | +| `x ILIKE 'a%'`, `x ~ 'regex'`, `x SIMILAR TO p` | Operators not supported | `LOWER(x) LIKE 'a%'`; PostgreSQL only: `similar_to(x, pattern[, escape])` | +| `a IS DISTINCT FROM b` | Operator form not supported | `is_distinct_from(a, b)` / `is_not_distinct_from(a, b)` (both databases); `isequal(a, b)` = null-safe equals | +| `ORDER BY x NULLS LAST` | Not supported | `ORDER BY x IS NULL, x` | +| `d + INTERVAL '1 day'` | No INTERVAL literals | `TIMESTAMPADD('SQL_TSI_DAY', 1, d)` | +| `CONCAT(a, b, c)` | `CONCAT` takes exactly 2 arguments | `a \|\| b \|\| c` (note: `\|\|` yields NULL if any operand is NULL — wrap with `COALESCE`) | +| `POSITION(a IN b)` | Unknown method | `LOCATE(a, b[, startIndex])` | +| `TRIM(BOTH ' ' FROM x)` | Not supported | `LTRIM(RTRIM(x))`; PostgreSQL only: `btrim(x)` | +| `COUNT(*) FILTER (WHERE c)` | No FILTER clause | `SUM(CASE WHEN c THEN 1 ELSE 0 END)` | +| `JOIN b USING (id)`, `NATURAL JOIN` | Every non-CROSS join requires `ON` | `JOIN b ON a.id = b.id` | +| `SUM(a \|\| b)`, `MIN(a & b)` | Aggregate arguments cannot directly contain `\|\|` or bitwise operators | Add parentheses: `SUM((a \|\| b))` | +| `DATE '2001-02-03'` | No typed literals | `{d '2001-02-03'}` or `CAST('2001-02-03' AS DATE)` | +| `{d'2001-02-03'}` (no space) | Parse error — the space after `{d`/`{ts` is part of the token | `{d '2001-02-03'}`, `{ts '2001-02-03 04:05:06'}` | +| `WITH cte (col1, col2) AS (...)` | No CTE column lists | Alias inside: `WITH cte AS (SELECT x AS col1 ...)` | +| `a < b < c` | Comparisons don't chain | `a < b AND b < c` | +| `PARAMETERS (D DATE)` | DATE/TIME/BOOLEAN not parameter types | Use `TIMESTAMP` (or `BIT` for boolean) | +| `GROUPING SETS / ROLLUP / CUBE / LATERAL / TABLESAMPLE` | Not supported | Restructure (UNION of grouped queries, etc.) | ----- -### **2. Calculated Columns** +### **2. Identifiers, Literals, and Reserved Words** -LabKey SQL allows you to create virtual columns within a query by using SQL expressions. These calculated columns are not stored in the database but are computed on the fly. +* **Identifiers**: double-quote names containing spaces/special characters or matching reserved words: `"Physical Exam"`. Escape an embedded `"` by doubling it. +* **String literals**: single quotes; escape `'` by doubling: `'Jim''s Item'`. No backslash escapes. +* **Date/time literals**: `{d '2001-02-03'}` and `{ts '2001-02-03 04:05:06'}` — JDBC escape syntax, **space after `{d`/`{ts` required**. +* **Booleans**: `TRUE`, `FALSE`. Special doubles: `CAST('Infinity' AS DOUBLE)`, `CAST('-Infinity' AS DOUBLE)`, `CAST('NaN' AS DOUBLE)`. +* **Reserved words** — these cannot be used as bare identifiers/aliases; double-quote them (`SELECT COUNT(*) AS "Count"`): -* **Syntax:** - `SELECT expression AS column_name FROM table` -* **Functionality:** - The syntax involves performing a calculation and then aliasing the result with a new column name using the `as` keyword. -* **Reserved words as aliases:** SQL function names (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`) and other SQL keywords are reserved and cannot be used as bare column aliases. Quote them or choose a different name: - ```sql - -- Wrong: COUNT(MouseId) AS Count → parse error - -- Right: COUNT(MouseId) AS "Count" - -- Right: COUNT(MouseId) AS TotalCount - ``` -* **Examples:** - * **Pulse Pressure:** To calculate pulse pressure from systolic and diastolic blood pressure values: - ```sql - PhysicalExam.systolicBP-PhysicalExam.diastolicBP as PulsePressure - ``` - * **BMI (Body Mass Index):** A more complex example that uses an intermediate query to calculate BMI from height and weight data: - ```sql - ROUND(weight_kg / (height_m * height_m), 2) AS BMI - ``` + `all, any, and, as, asc, avg, between, both, case, class, count, delete, desc, distinct, elements, else, empty, end, escape, except, exists, false, fetch, from, full, group, having, in, indices, inner, insert, intersect, into, is, join, leading, left, like, limit, max, member, min, new, not, null, of, on, or, order, outer, right, select, set, some, stddev, sum, trailing, then, true, union, update, user, versioned, when, where` + +* Do not end statements with `;` (produces a warning) and do not submit multiple statements. ----- -### **3. Pivoting** +### **3. Query Structure and Clause Rules** -A `PIVOT` query helps you summarize and re-visualize data by transforming rows into columns. +* **SELECT list**: `expr [AS] alias`. Alias every expression column — unaliased expressions get auto-names (`Expression1`, …) plus a warning. Duplicate output names are an error (`Duplicate column 'x'`). `*` and `table.*` cannot be aliased. A scalar subquery must return exactly one column. `SELECT 1 AS x` with no FROM is allowed. +* **FROM/JOIN**: `INNER | LEFT | RIGHT | FULL [OUTER] JOIN ... ON cond` and `CROSS JOIN` (no ON). Comma joins and nested parenthesized joins are supported. Subqueries in FROM should be aliased (warning otherwise). An unqualified column name found in two FROM tables is an error (`Ambiguous field`). +* **WHERE**: standard comparisons `= <> != < <= > >=`, `[NOT] IN (list | subquery)`, `[NOT] BETWEEN a AND b`, `[NOT] LIKE p [ESCAPE e]`, `IS [NOT] NULL`, `EXISTS (subquery)`, `ANY/SOME/ALL (subquery)`. +* **GROUP BY**: expressions only — no ordinals, no constants. **HAVING requires GROUP BY or an aggregate in the SELECT list** (an aggregate appearing only in HAVING is rejected). +* **ORDER BY**: column names, aliases, expressions, or ordinals (`ORDER BY 1`); `ASC`/`DESC`. **When a query is used as a subquery or saved and wrapped by the server (common), its ORDER BY is IGNORED unless LIMIT is also present.** Prefer sorting via the client API / grid view; if SQL sorting is needed, add `LIMIT`. To sort by an expression, put it in the SELECT list (optionally `@hidden`) and sort by its alias. +* **LIMIT**: `LIMIT ` only, placed after ORDER BY. Applies to the whole UNION when used at the end of one. +* **Set operations**: `UNION`, `UNION ALL`, `INTERSECT`, `EXCEPT`. All terms need the same column count and compatible types; result column names come from the first term. A trailing ORDER BY may reference only output column names or ordinals. Parenthesized sub-statements are allowed. +* **Division**: `/` warns unless the divisor is guarded — write `dividend / NULLIF(divisor, 0)`. Integer ÷ integer truncates; cast one operand: `CAST(a AS DOUBLE) / b`. -* **Syntax for PIVOT...BY Query:** - A pivot query is a `SELECT` statement specifying how to pivot and group columns. The basic syntax is `PIVOT [aggregating_column] BY [pivoting_column]`. -* **PIVOT...BY...IN Syntax:** - You can use an `IN` clause to specify a fixed set of column names to pivot. This is more efficient. - ```sql - PIVOT new_column_name BY pivoting_column IN ('value1', 'value2') - ``` - Note that pivot column names are case-sensitive. You may need to use `LOWER()` or `UPPER()` in your query to work around this issue. - * **Pivoting by Two Columns:** - Two levels of `PIVOT` are not directly supported. However, you can achieve a similar result by concatenating the two values together and pivoting on that "calculated" column. - ```sql - SELECT - Run.SampleCondition || ' ' || PeakLabel AS ConditionPeak, - AVG(Data.PercTimeCorrArea) AS AvgPercTimeCorrArea - FROM Data - GROUP BY Run.SampleCondition || ' ' || PeakLabel - PIVOT AvgPercTimeCorrArea BY ConditionPeak - ``` +----- + +### **4. Lookups (Foreign-Key Dot Traversal) and Joins** + +Lookup columns (foreign keys) can be traversed with dot notation instead of writing a JOIN. Follow the FK column with the target column name: + +```sql +-- CreatedBy is a lookup to the Users table: +SELECT c.Name, c.CreatedBy.DisplayName AS Creator +FROM core.Containers c +``` + +Multi-hop traversal works (`Issues.AssignedTo.DisplayName`). Only reference tables in the FROM clause or reachable through lookups — referencing another table's columns directly is an error. + +**Study datasets**: every dataset has a `Datasets` column joining to other datasets by subject(/visit): + +```sql +SELECT a.MouseId, a.Datasets."DEM-1".DEMsex +FROM "APX-1" a +``` + +**Joining across folders** (see also section 12): + +```sql +SELECT d.Language, l.TranslatorName +FROM Demographics d JOIN "/Other/Folder".lists.Languages l ON d.Language = l.Language +``` + +`IFDEFINED(ColumnName)` references a column that may not exist — it evaluates as NULL instead of failing, useful over PIVOT results or variable assay schemas. ----- -### **4. Cross-Folder Queries** +### **5. Aggregate Functions** -You can write queries that access data from different folders within the LabKey Server instance, allowing for data integration across projects. +Available on both databases: -* **Syntax:** - `SELECT * FROM Project."folder_path".schema.table` -* **Functionality:** - The folder path is a dot-delimited string that specifies the location of the table, including the project name. The user must have "Reader" permissions in each folder referenced in the query. -* **Example:** - ```sql - SELECT - p.ParticipantID, - ROUND(AVG(p.Temp_C), 1) AS AverageTemp - FROM Project."Tutorials/Demo/".study."Physical Exam" p - GROUP BY p.ParticipantID - ``` +* `COUNT(*)`, `COUNT(expr)`, `COUNT(DISTINCT expr)` +* `SUM(expr)`, `MIN(expr)`, `MAX(expr)`, `AVG(expr)` +* `GROUP_CONCAT([DISTINCT] expr [, separator])`: comma-separated values of the group; e.g. `GROUP_CONCAT(DISTINCT Category, ';')`. (On MS SQL Server requires a separately-installed function and cannot be used in a sub-select.) +* `STDDEV(expr)`, `STDDEV_POP(expr)`, `VARIANCE(expr)`, `VAR_POP(expr)`, `STDERR(expr)` +* `MEDIAN(expr)` + +PostgreSQL only: `BOOL_AND, BOOL_OR, EVERY, BIT_AND, BIT_OR, MODE, STDDEV_SAMP, VAR_SAMP` and two-argument regression aggregates `CORR(Y,X), COVAR_POP, COVAR_SAMP, REGR_AVGX, REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE, REGR_SXX, REGR_SXY, REGR_SYY`. + +Rules: `DISTINCT` is allowed only in `COUNT` and `GROUP_CONCAT`. No `FILTER` clause, no `OVER`, no `ARRAY_AGG`. To nest `||` or bitwise operators inside an aggregate, parenthesize: `MIN((a || b))`. ----- -### **5. Parameterized Queries** +### **6. Scalar Functions (Both Databases)** -LabKey SQL supports **parameterized queries** to improve security and reusability. +#### Mathematical +`abs(v)`, `acos(v)`, `asin(v)`, `atan(v)`, `atan2(v1,v2)`, `ceiling(v)`, `cos(r)`, `cot(r)`, `degrees(r)`, `exp(n)`, `floor(v)`, `log(n)` (natural), `log10(n)`, `mod(dividend, divider)`, `pi()`, `power(base, exp)`, `radians(d)`, `rand([seed])`, `round(v[, precision])`, `sign(v)`, `sin(v)`, `sqrt(v)`, `tan(v)`, `truncate(v, precision)` (may require `CAST(v AS NUMERIC)`) -* **Syntax:** - `PARAMETERS(param1 type, param2 type DEFAULT value) SELECT * FROM table WHERE column = param1` -* **Functionality:** - The `PARAMETERS` keyword declares parameters that can be passed into the query. If a DEFAULT is - not specified, the value will default to NULL. -* **Example:** A query with two parameters, `MinTemp` and `MinWeight`: - ```sql - PARAMETERS(MinTemp double, MinWeight double DEFAULT 0.0) - SELECT - ParticipantID, - temperature_C, - weight_kg - FROM PhysicalExam - WHERE temperature_C >= MinTemp AND weight_kg >= MinWeight - ``` +#### String +`concat(a, b)` (exactly 2 args; prefer `||`), `lcase(s)`/`lower(s)`, `ucase(s)`/`upper(s)`, `left(s, n)`, `right(s, n)`, `length(s)`, `locate(substr, s[, start])`, `ltrim(s)`, `rtrim(s)`, `repeat(s, count)`, `startswith(s, prefix)`, `substring(s, start[, length])` (1-based) + +#### Date and Time +* `curdate()`, `curtime()`, `now()` +* `year(d)`, `quarter(d)`, `month(d)`, `monthname(d)`, `week(d)`, `dayofyear(d)`, `dayofmonth(d)`, `dayofweek(d)`, `hour(t)`, `minute(t)`, `second(t)` +* `timestampadd(interval, n, ts)` — interval is a quoted constant, one of `'SQL_TSI_FRAC_SECOND'`, `'SQL_TSI_SECOND'`, `'SQL_TSI_MINUTE'`, `'SQL_TSI_HOUR'`, `'SQL_TSI_DAY'`, `'SQL_TSI_WEEK'`, `'SQL_TSI_MONTH'`, `'SQL_TSI_QUARTER'`, `'SQL_TSI_YEAR'` (the `SQL_TSI_` prefix may be omitted: `'DAY'`). +* `timestampdiff(interval, ts1, ts2)` — same constants, **but on PostgreSQL only `'SQL_TSI_SECOND'`, `'SQL_TSI_MINUTE'`, `'SQL_TSI_HOUR'`, `'SQL_TSI_DAY'` work**; YEAR/MONTH/WEEK/QUARTER fail at execution time. For those use: +* `age(d1, d2)` (years), `age(d1, d2, interval)` with `'SQL_TSI_DAY' | 'SQL_TSI_MONTH' | 'SQL_TSI_YEAR'`, `age_in_years(d1, d2)`, `age_in_months(d1, d2)`, `age_in_days(d1, d2)` + +#### Conditional and Utility +`coalesce(v1, ..., vN)`, `nullif(a, b)` (NULL if a=b, else a — use for divide-by-zero guards), `ifnull(test, default)`, `isequal(a, b)` (true when equal or both NULL), `is_distinct_from(a, b)`, `is_not_distinct_from(a, b)`, `greatest(a, b, ...)`, `least(a, b, ...)`, `isnumeric(expr)`, `ifdefined(col)`, `CASE [operand] WHEN ... THEN ... [ELSE ...] END` + +#### LabKey Extensions +`userid()`, `username()`, `ismemberof(groupid)`, `contextPath()`, `folderName()`, `folderPath()`, `moduleProperty('module','property')`, `javaConstant('class.FIELD')`, `version()`, `overlaps(start1, end1, start2, end2)` (PostgreSQL only) ----- -### **6. Metadata Annotations** +### **7. PostgreSQL-Only Scalar Functions** -LabKey SQL allows you to directly annotate your SQL statements to override how column metadata is displayed in the LabKey interface. +Only when the server runs PostgreSQL (the common case). Passed through natively: -* **Syntax:** - `SELECT column_name @annotation FROM table` -* **Functionality:** - Annotations control the display of a column without changing the underlying data. -* **Examples:** - * **Hiding a Column:** - ```sql - SELECT ratio @hidden, log(ratio) as log_ratio - ``` - * **Setting a Title and Format:** - ```sql - SELECT 10/7.0 AS Num @title='Calculated Number' @Format='0.00' - ``` +`ascii`, `btrim(s[,chars])`, `char_length`, `character_length`, `chr(code)`, `concat_ws(sep, v1, ...)` (variadic, skips NULLs), `decode`, `encode`, `initcap`, `lpad(s,n[,fill])`, `rpad(s,n[,fill])`, `md5`, `octet_length`, `quote_ident`, `quote_literal`, `regexp_replace(s, pattern, replacement[, flags])`, `replace(s, match, replacement)`, `similar_to(s, pattern[, escape])`, `split_part(s, delim, n)`, `strpos(s, sub)`, `substr(s, from[, count])`, `to_ascii`, `to_hex`, `translate(s, from, to)`, `to_char(v, format)`, `to_date(text, format)`, `to_timestamp(text, format)`, `to_number(text, format)`, `string_to_array`, `unnest` + +(MS SQL Server-only equivalents exist — `charindex`, `len`, `patindex`, `replicate`, `stuff`, etc. — only relevant for premium MSSQL deployments.) ----- -### **7. Container Filters** - -In addition to targeting a container by its path, LabKey SQL supports container filters to alter the scope -of a query. Annotate tables in the FROM clause with an optional container filter. Syntax: - -SELECT * FROM Issues [ContainerFilter='CurrentAndSubfolders'] alias - -Possible values include: -- AllFolders -- AllInProject -- AllInProjectPlusShared -- Current -- CurrentAndFirstChildren -- CurrentAndParents -- CurrentAndSubfolders -- CurrentAndSubfoldersPlusShared -- CurrentPlusProject -- CurrentPlusProjectAndShared. - -### **8. Available Methods** - -Here is a summary of the available functions and methods in LabKey SQL. - -#### **Mathematical Functions** - -* `abs(value)`: Returns the absolute value. -* `acos(value)`: Returns the arc cosine. -* `asin(value)`: Returns the arc sine. -* `atan(value)`: Returns the arc tangent. -* `atan2(value1, value2)`: Returns the arctangent of the quotient. -* `ceiling(value)`: Rounds the value up. -* `cos(radians)`: Returns the cosine. -* `cot(radians)`: Returns the cotangent. -* `degrees(radians)`: Returns degrees. -* `exp(n)`: Returns Euler's number 'e' raised to the nth power. -* `floor(value)`: Rounds down. -* `log(n)`: Returns the natural logarithm. -* `log10(n)`: Returns the base 10 logarithm. -* `mod(dividend, divider)`: Returns the remainder. -* `pi()`: Returns the value of pi. -* `power(base, exponent)`: Returns the base raised to the power of the exponent. -* `radians(degrees)`: Returns the radians. -* `rand()`, `rand(seed)`: Returns a random number. -* `round(value, precision)`: Rounds to the specified decimal places. -* `sign(value)`: Returns the sign of the value. -* `sin(value)`: Returns the sine. -* `sqrt(value)`: Returns the square root. -* `tan(value)`: Returns the tangent. -* `truncate(numeric value, precision)`: Truncates the numeric value. - -#### **String Functions** - -* `concat(value1, value2)`: Concatenates two values. -* `lcase(string)`, `lower(string)`: Converts to lower case. -* `left(string, integer)`: Returns the left side of the string. -* `length(string)`: Returns the length. -* `locate(substring, string, [startIndex])`: Returns the location of a substring. -* `ltrim(string)`: Trims white space from the left. -* `repeat(string, count)`: Repeats the string. -* `rtrim(string)`: Trims white space from the right. -* `startswith(string, prefix)`: Tests if a string starts with a prefix. -* `substring(string, start, length)`: Returns a portion of the string. -* `ucase(string)`, `upper(string)`: Converts to upper case. - -#### **Date and Time Functions** - -* `age(date1, date2, [interval])`: Supplies the difference in age. -* `age_in_days(date1, date2)`: Returns age in days. -* `age_in_months(date1, date2)`: Returns age in months. -* `age_in_years(date1, date2)`: Returns age in years. -* `curdate()`, `curtime()`: Returns the current date/time. -* `dayofmonth(date)`: Returns the day of the month. -* `dayofweek(date)`: Returns the day of the week. -* `dayofyear(date)`: Returns the day of the year. -* `hour(time)`, `minute(time)`, `second(time)`: Return time components. -* `month(date)`, `monthname(date)`: Return month values. -* `now()`: Returns the system date and time. -* `quarter(date)`: Returns the yearly quarter. -* `timestampadd(interval, number, timestamp)`: Adds an interval. -* `timestampdiff(interval, ts1, ts2)`: Finds the difference between timestamps. -* `week(date)`, `year(date)`: Return week and year values. - -#### **Conditional and Utility Functions** - -* `coalesce(v1,...,vN)`: Returns the first non-null value. -* `greatest(a, b, c, ...)`: Returns the greatest value. -* `ifdefined(column_name)`: References columns that may not exist. -* `ifnull(testValue, defaultValue)`: Returns a default value if the test value is null. -* `isequal(a,b)`: Returns true if `a` equals `b` or if both are `NULL`. -* `least(a, b, c, ...)`: Returns the smallest value. - -#### **LabKey SQL Extensions** - -* `contextPath()`, `folderName()`, `folderPath()`: Return path information. -* `ismemberof(groupid)`: Checks if a user is a member of a group. -* `javaConstant(fieldName)`: Provides access to Java static final variables. -* `moduleProperty(module name, property name)`: Returns a module property. -* `overlaps(START1, END1, START2, END2)`: Tests for overlapping time intervals (PostgreSQL only). -* `userid()`, `username()`: Return user information. -* `version()`: Returns the current schema version. +### **8. WITH (Common Table Expressions)** + +```sql +WITH AllDemo AS ( + SELECT * FROM "/Studies/Study A/".study.Demographics + UNION + SELECT * FROM "/Studies/Study B/".study.Demographics +) +SELECT ParticipantId FROM AllDemo +``` + +Rules: +* `WITH name AS (SELECT ...)` — **no column list** after the name. +* Multiple CTEs allowed; each may reference earlier ones. +* Recursive CTEs are supported (no `RECURSIVE` keyword needed). The recursive form is `anchor-query UNION ALL recursive-query`; the **first (anchor) branch must not reference the CTE**, and the CTE may be referenced **only once** in the recursive branch. +* **Warning**: a non-terminating recursive CTE can hang the server (PostgreSQL raises no error) — always ensure termination. + +```sql +PARAMETERS ( Source VARCHAR DEFAULT NULL ) +WITH Derivations AS ( + SELECT Item, Parent FROM Items WHERE Parent = Source + UNION ALL + SELECT i.Item, i.Parent FROM Items i INNER JOIN Derivations p ON i.Parent = p.Item +) +SELECT * FROM Derivations +``` ----- -### **9. JSON and JSONB Operators and Functions (PostgreSQL Only)** - -LabKey SQL supports PostgreSQL JSON and JSONB operators and functions for working with JSON data stored in columns. These are **not available on MS SQL Server**. See the [PostgreSQL docs](https://www.postgresql.org/docs/14/functions-json.html) for detailed usage. - -#### **Operators via `json_op`** - -Native PostgreSQL operator syntax (`->`, `->>`, etc.) cannot be used directly in LabKey SQL. Instead, use the `json_op` pass-through function with three arguments: the left operand, the operator as a string, and the right operand. - -* **Supported operators:** `->`, `->>`, `#>`, `#>>`, `@>`, `<@`, `?`, `?|`, `?&`, `||`, `-`, `#-` -* **Syntax:** `json_op(left_operand, 'operator', right_operand)` -* **Examples:** - * **Extract by key (as JSON):** Get a nested value from a JSONB column: - ```sql - SELECT json_op(metadata, '->', 'name') AS name_json FROM samples - ``` - * **Extract by key (as text):** Get the text value: - ```sql - SELECT json_op(metadata, '->>', 'name') AS name_text FROM samples - ``` - * **Containment check:** Filter rows where JSONB contains a given structure: - ```sql - SELECT * FROM samples WHERE json_op(metadata, '@>', parse_jsonb('{"status":"active"}')) - ``` - * **Key existence check:** - ```sql - SELECT * FROM samples WHERE json_op(metadata, '?', 'name') - ``` - -#### **Conversion / Parsing Functions** - -* `parse_json(text)`, `parse_jsonb(text)`: Cast a text value to JSON or JSONB. Use instead of `::jsonb` or `CAST(... AS JSONB)`. - ```sql - SELECT parse_jsonb('{"a":1, "b":null}') - ``` -* `to_json(value)`, `to_jsonb(value)`: Convert a value to JSON/JSONB. Text values become a single JSON string. -* `array_to_json(array)`: Convert an array to JSON. -* `row_to_json(value)`: Convert a scalar row to JSON. **Note:** Does not support converting an entire table to JSON; use `to_jsonb()` instead. +### **9. VALUES (Constant Tables)** -#### **Builder Functions** +```sql +SELECT t.column1 AS Id, t.column2 AS Name +FROM (VALUES (1, 'one'), (2, 'two'), (3, 'three')) AS t +``` -* `json_build_array(...)`, `jsonb_build_array(...)`: Build a JSON array from arguments. - ```sql - SELECT jsonb_build_array(1, 'two', 3.0) - ``` -* `json_build_object(...)`, `jsonb_build_object(...)`: Build a JSON object from key/value arguments. - ```sql - SELECT jsonb_build_object('name', sample_name, 'type', sample_type) FROM samples - ``` -* `json_object(text_array)`, `jsonb_object(text_array)`: Build a JSON object from a text array. +* The table alias (`AS t`) is **required**; column names are always `column1`, `column2`, … and cannot be declared in the VALUES clause — rename them in the outer SELECT. +* Rows must have equal length and compatible types; only constants/parameters (no column references or subqueries). -#### **Query and Extraction Functions** +----- -* `json_array_length(json)`, `jsonb_array_length(jsonb)`: Return the length of the outermost JSON array. -* `json_each(json)`, `jsonb_each(jsonb)`: Expand the outermost JSON object into key/value pairs. **Note:** Only scalar function usage is supported, not the table-returning version. - ```sql - SELECT json_each('{"a":"foo", "b":"bar"}') AS Value - ``` -* `json_each_text(json)`, `jsonb_each_text(jsonb)`: Like `json_each` but values are returned as text. Only scalar usage supported. -* `json_extract_path(json, ...)`, `jsonb_extract_path(jsonb, ...)`: Return the JSON value at the given path. - ```sql - SELECT jsonb_extract_path(metadata, 'address', 'city') FROM samples - ``` -* `json_extract_path_text(json, ...)`, `jsonb_extract_path_text(jsonb, ...)`: Return the value at the given path as text. -* `json_object_keys(json)`, `jsonb_object_keys(jsonb)`: Return the keys of the outermost JSON object. -* `json_array_elements(json)`, `jsonb_array_elements(jsonb)`: Expand a JSON array into a set of values. -* `json_array_elements_text(json)`, `jsonb_array_elements_text(jsonb)`: Expand a JSON array into a set of text values. +### **10. PIVOT** -#### **Type Inspection and Cleanup Functions** +A pivot query summarizes and rotates a **grouped** query: `PIVOT aggCol [, aggCol...] BY pivotColumn [IN (...)]`. -* `json_typeof(json)`, `jsonb_typeof(jsonb)`: Return the type of the outermost JSON value (e.g., `"object"`, `"array"`, `"string"`, `"number"`). -* `json_strip_nulls(json)`, `jsonb_strip_nulls(jsonb)`: Remove all null-valued keys from a JSON object. +```sql +SELECT ParticipantId, Visit, AVG(Score) AS AvgScore +FROM Results +GROUP BY ParticipantId, Visit +PIVOT AvgScore BY Visit IN ('V1', 'V2', 'V3') +``` -#### **Modification Functions** +Hard rules (violations are errors): +* The query **must have a GROUP BY clause**, and the BY column's expression must appear in GROUP BY **exactly as written**. +* Each pivoted column (`AvgScore` above) must be an aggregate in the SELECT list. +* Any additional non-pivoted, non-grouping aggregate column may only use `SUM`, `MIN`, `MAX`, or `COUNT` — not AVG/GROUP_CONCAT/stddev. +* `IN (...)` takes constants with optional aliases (`IN ('a' AS ColA, 'b' ColB)`) **or a subselect** (`IN (SELECT DISTINCT ...)`). Omitting IN computes distinct values automatically, but **a parameterized query requires an explicit IN list**. +* Pivot output columns are named `value::aggAlias` (e.g. `V1::AvgScore`). Names are case-insensitive; duplicate pivot values differing only by case are an error — normalize with `LOWER()`/`UPPER()` in the query. +* Two-level pivots are not supported; concatenate the two values into one column and pivot on that. -* `jsonb_insert(jsonb, path, new_value)`: Insert a value at a given path within a JSONB object. -* `jsonb_pretty(jsonb)`: Format a JSONB value as indented text. -* `jsonb_set(jsonb, path, new_value)`: Set the value at a given path. Strict: returns NULL on NULL input. -* `jsonb_set_lax(jsonb, path, new_value, null_behavior)`: Like `jsonb_set` but not strict. The `null_behavior` argument must be one of: `'raise_exception'`, `'use_json_null'`, `'delete_key'`, or `'return_target'`. +----- -#### **Path Query Functions** +### **11. Parameterized Queries** -* `jsonb_path_exists(jsonb, path)`, `jsonb_path_exists_tz(...)`: Check whether the JSON path returns any item. The `_tz` variant is timezone-aware. -* `jsonb_path_match(jsonb, path)`, `jsonb_path_match_tz(...)`: Return the result of a JSON path predicate check. -* `jsonb_path_query(jsonb, path)`, `jsonb_path_query_tz(...)`: Return all items matched by the JSON path. -* `jsonb_path_query_array(jsonb, path)`, `jsonb_path_query_array_tz(...)`: Return matched items as an array. -* `jsonb_path_query_first(jsonb, path)`, `jsonb_path_query_first_tz(...)`: Return the first matched item. +```sql +PARAMETERS (MinTemp DOUBLE, MinWeight DOUBLE DEFAULT 0.0) +SELECT ParticipantID, temperature_C, weight_kg +FROM PhysicalExam +WHERE temperature_C >= MinTemp AND weight_kg >= MinWeight +``` -#### **Not Supported** +* Allowed types: `BIGINT, BIT, CHAR, DECIMAL, DOUBLE, FLOAT, INTEGER, LONGVARCHAR, NUMERIC, REAL, SMALLINT, TIMESTAMP, TINYINT, VARCHAR` (NUMERIC accepts precision/scale). **DATE, TIME, and BOOLEAN are not allowed** — use TIMESTAMP / BIT. +* `DEFAULT` value must be a constant; without DEFAULT the parameter is required (NULL when unset via the API). +* A parameter name shadows an unqualified column of the same name — qualify the column (`R.X`) to disambiguate. +* Values are passed via the client API (e.g. `query.param.MinTemp`). -The following functions are **not supported** in LabKey SQL: -`json_populate_record`, `jsonb_populate_record`, `json_populate_recordset`, `jsonb_populate_recordset`, `json_to_record`, `jsonb_to_record`, `json_to_recordset`, `jsonb_to_recordset`. +----- + +### **12. Cross-Folder Queries** + +Prefix the schema with a quoted folder path (user needs Reader permission in each referenced folder): + +```sql +SELECT p.ParticipantID, ROUND(AVG(p.Temp_C), 1) AS AverageTemp +FROM "/Tutorials/Demo".study."Physical Exam" p +GROUP BY p.ParticipantID +``` + +`Project` refers to the current project root: `Project."SubFolder".lists.MyList`. + +----- + +### **13. Container Filters** + +Annotate a table in the FROM clause to broaden the query scope: + +```sql +SELECT * FROM Issues [ContainerFilter='CurrentAndSubfolders'] i +``` + +Values: `AllFolders`, `AllInProject`, `AllInProjectPlusShared`, `Current`, `CurrentAndFirstChildren`, `CurrentAndParents`, `CurrentAndSubfolders`, `CurrentAndSubfoldersPlusShared`, `CurrentPlusProject`, `CurrentPlusProjectAndShared`. + +----- -#### **Quick Reference for Writing LabKey SQL with JSON** +### **14. Metadata Annotations** -When writing LabKey SQL queries that work with JSON columns: +Column annotations after a select expression override display metadata: -1. **Always use `json_op()` for operators** — never use raw PostgreSQL operator syntax like `->` or `->>`. Wrap them: `json_op(col, '->>', 'key')`. -2. **Use `parse_jsonb()` to create JSONB literals** — there is no `::jsonb` cast in LabKey SQL. Write `parse_jsonb('{"key":"value"}')`. -3. **Use `jsonb_extract_path_text()` for nested field access** — this is often the clearest way to extract a deeply nested text value: `jsonb_extract_path_text(col, 'level1', 'level2', 'field')`. -4. **Use `jsonb_build_object()` to construct JSON** — for building JSON from column values: `jsonb_build_object('id', rowid, 'name', label)`. -5. **Check database type first** — these functions only work on PostgreSQL. If the target server may use MS SQL Server, do not use them. -6. **The `validateSQL` MCP tool can verify syntax** — use it to check JSON function calls before the user saves a query. +```sql +SELECT ratio @hidden, + 10/7.0 AS Num @title='Calculated Number' @format='0.00' +``` + +Recognized: `@title`, `@format`, `@hidden`, `@concept`, `@nolookup`, `@preservetitle`. Value syntax: `@title='x'` or `@title('x')`. ----- -### **10. Array Functions (PostgreSQL Only)** +### **15. CAST** + +`CAST(expression AS type)`. LabKey SQL does **not** reliably check argument types at parse time — type errors surface at execution as a generic error, so cast proactively (e.g., a date stored as VARCHAR passed to a date function). -LabKey SQL supports a set of array construction and comparison functions. These are **not available on MS SQL Server**. They route through dialect-specific SQL generation rather than passing through directly to PostgreSQL. +* Target types: `TINYINT, SMALLINT, INTEGER, BIGINT, REAL, FLOAT, DOUBLE, NUMERIC, DECIMAL, BOOLEAN, BIT, CHAR, VARCHAR, LONGVARCHAR, DATE, TIME, TIMESTAMP, GUID, BINARY, VARBINARY, LONGVARBINARY`. Precision/scale on NUMERIC/DECIMAL: `CAST(n AS NUMERIC(10,2))`; length on CHAR/VARCHAR. +* Common patterns: `CAST(stringCol AS TIMESTAMP)`, `CAST(numericCol AS VARCHAR) || ' units'`, `CAST(numerator AS DOUBLE) / denominator` (avoid integer-division truncation). -#### **Construction** +----- -* `ARRAY[elem, ...]` → ARRAY: builds an array from zero or more elements of any type. Equivalent to PostgreSQL `ARRAY[...]`. -* `TEXT_ARRAY[elem, ...]` → TEXT[]: like `ARRAY[]` but casts the result to `TEXT[]`. +### **16. JSON and JSONB Operators and Functions (PostgreSQL Only)** -#### **Element Membership** +Not available on MS SQL Server. See the [PostgreSQL docs](https://www.postgresql.org/docs/14/functions-json.html) for semantics. -* `array_contains_element(array, element)` → BOOLEAN: true if the element appears in the array. Equivalent to PostgreSQL `element = ANY(array)`. Use `NOT array_contains_element(arr, val)` for the negative — there is no separate "does not contain" function. +#### Operators via `json_op` -#### **Array-vs-Array Comparisons** +Native operator syntax (`->`, `->>`, etc.) cannot be used. Use the `json_op` pass-through function: `json_op(left_operand, 'operator', right_operand)`. -* `array_contains_all(array_a, array_b)` → BOOLEAN: true if every element of `array_b` is in `array_a`. Equivalent to PostgreSQL `array_a @> array_b`. -* `array_contains_any(array_a, array_b)` → BOOLEAN: true if at least one element of `array_b` is in `array_a`. Equivalent to PostgreSQL `array_a && array_b`. -* `array_contains_none(array_a, array_b)` → BOOLEAN: true if no element of `array_b` is in `array_a`. -* `array_is_same(array_a, array_b)` → BOOLEAN: unordered set equality — both arrays contain exactly the same elements regardless of order. Use `NOT array_is_same(...)` for the negative. +* Supported operators: `->`, `->>`, `#>`, `#>>`, `@>`, `<@`, `?`, `?|`, `?&`, `||`, `-`, `#-` +* Examples: + ```sql + SELECT json_op(metadata, '->>', 'name') AS name_text FROM samples + SELECT * FROM samples WHERE json_op(metadata, '@>', parse_jsonb('{"status":"active"}')) + SELECT * FROM samples WHERE json_op(metadata, '?', 'name') + ``` -#### **Emptiness** +#### Conversion / Parsing +* `parse_json(text)`, `parse_jsonb(text)`: cast text to JSON/JSONB (instead of `::jsonb` / `CAST(... AS JSONB)`). +* `to_json(value)`, `to_jsonb(value)`; `array_to_json(array)`; `row_to_json(value)` (scalar rows only — use `to_jsonb()` for whole-row). -* `array_is_empty(array)` → BOOLEAN: true if the array has no elements. The argument must be of type ARRAY; a type mismatch is caught at query-parse time. +#### Builders +* `json_build_array(...)`, `jsonb_build_array(...)`; `json_build_object(k1, v1, ...)`, `jsonb_build_object(...)`; `json_object(text_array)`, `jsonb_object(text_array)`. -#### **Not Supported** +#### Query and Extraction +* `json_array_length`, `jsonb_array_length`; `json_extract_path(json, ...)`, `jsonb_extract_path(...)`; `json_extract_path_text(...)`, `jsonb_extract_path_text(...)` (clearest for nested text values); `json_object_keys`, `jsonb_object_keys`; `json_array_elements[_text]`, `jsonb_array_elements[_text]`; `json_each`, `jsonb_each`, `json_each_text`, `jsonb_each_text` (**scalar usage only** — not usable as table sources in FROM). -`array_length`, `array_append`, `array_prepend`, `array_cat`, `array_remove`, `array_replace`, `array_position`, `array_to_string`, and subscript access (`arr[n]`) are not available in LabKey SQL. +#### Type Inspection, Cleanup, Modification +* `json_typeof`, `jsonb_typeof`; `json_strip_nulls`, `jsonb_strip_nulls`; `jsonb_insert(jsonb, path, val)`; `jsonb_pretty(jsonb)`; `jsonb_set(jsonb, path, val)` (strict — NULL in, NULL out); `jsonb_set_lax(jsonb, path, val, null_behavior)` where null_behavior ∈ `'raise_exception' | 'use_json_null' | 'delete_key' | 'return_target'`. -### **11. CAST** +#### Path Queries +* `jsonb_path_exists`, `jsonb_path_match`, `jsonb_path_query`, `jsonb_path_query_array`, `jsonb_path_query_first` — each with a timezone-aware `_tz` variant. + +#### Not Supported +`json_populate_record`, `jsonb_populate_record`, `json_populate_recordset`, `jsonb_populate_recordset`, `json_to_record`, `jsonb_to_record`, `json_to_recordset`, `jsonb_to_recordset`. + +----- -Use `CAST(expression AS type)` to convert a value from one type to another. **LabKey SQL's validator does not -reliably detect type mismatches** — errors only surface as runtime exceptions, so use CAST proactively whenever -there is ambiguity (e.g., a date stored as VARCHAR passed to a date/time function). +### **17. Array Functions (PostgreSQL Only)** -**Important:** PostgreSQL's `::type` shorthand (e.g., `col::integer`) is **not** supported in LabKey SQL. Always use `CAST()`. +Not available on MS SQL Server. -* **Supported target types:** - * Integers: `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT` - * Floating point: `REAL`, `FLOAT`, `DOUBLE` - * Fixed-point: `NUMERIC`, `DECIMAL` - * Boolean: `BOOLEAN`, `BIT` - * String: `CHAR`, `VARCHAR`, `LONGVARCHAR` - * Date/time: `DATE`, `TIME`, `TIMESTAMP` - * Other: `GUID` +#### Construction +* `ARRAY[elem, ...]` — builds an array. **No space between `ARRAY` and `[`.** +* `TEXTARRAY[elem, ...]` — like `ARRAY[]` but cast to `TEXT[]`. (One word — `TEXT_ARRAY` is wrong.) -* **Examples:** - * **String column used as a number:** - ```sql - CAST(stringCol AS DOUBLE) - ``` - * **String column used as a timestamp:** - ```sql - CAST(stringCol AS TIMESTAMP) - ``` - * **Number formatted as a string (e.g., for concatenation):** - ```sql - CAST(numericCol AS VARCHAR) || ' units' - ``` - * **Ensuring integer division doesn't truncate:** - ```sql - CAST(numerator AS DOUBLE) / denominator - ``` +#### Membership and Comparison +* `array_contains_element(array, elem)` — true if elem in array (`= ANY`). Negate with `NOT array_contains_element(...)`. +* `array_contains_all(a, b)` — every element of b is in a (`a @> b`). +* `array_contains_any(a, b)` — at least one element of b is in a (`a && b`). +* `array_contains_none(a, b)` — no element of b is in a. +* `array_is_same(a, b)` — unordered set equality. +* `array_is_empty(a)` — argument must be an ARRAY type (checked at parse time). + +#### Not Supported +`array_agg`, `array_length`, `array_append`, `array_prepend`, `array_cat`, `array_remove`, `array_replace`, `array_position`, `array_to_string`, and subscript access (`arr[n]`). + +----- + +### **18. Database Portability Notes** + +Most LabKey servers run PostgreSQL; some premium deployments run MS SQL Server. If the target may be MSSQL: + +* Avoid PostgreSQL-only features: JSON functions (§16), arrays (§17), `similar_to`/`regexp_replace`/`to_char`-family (§7), `overlaps()`, and the PG-only aggregates (§5). +* `GROUP_CONCAT` on MSSQL requires a separately-installed function and cannot appear in a sub-select; when missing, it returns a literal `''` string instead of data. +* String comparison and grouping are case-sensitive on PostgreSQL, case-insensitive on MSSQL. +* `timestampdiff` interval limits differ (§6) — the `age*` functions are portable. + +----- -* **Quick Reference:** - 1. **VARCHAR passed to a date/time function** — wrap it: `CAST(col AS TIMESTAMP)`. - 2. **Integer arithmetic truncates unexpectedly** — cast one operand to `DOUBLE`. - 3. **Do not rely on `validateSQL` for type errors** — mismatches only appear at runtime. \ No newline at end of file +### **19. Common Error Messages → Fixes** + +Many parse errors now include an inline suggestion (e.g. `Syntax error near 'OFFSET' ... OFFSET is not supported. Use LIMIT n...` or `Unknown method TRIM. Use LTRIM(RTRIM(x)).`). When a suggestion is present, apply it directly — it is dialect-appropriate for the server that produced it. + +| Error | Likely cause / fix | +| --- | --- | +| `Syntax error near 'OFFSET'` / near `'('` after OVER | Unsupported OFFSET / window function — see §1 | +| `Expression in Group By clause must not be a constant` | `GROUP BY 1` — repeat the expression instead | +| `Syntax error near 'DISTINCT'` | `SUM(DISTINCT ...)` or `IS DISTINCT FROM` — see §1 | +| `Could not resolve column: CURRENT_DATE` | Use `CURDATE()` / `NOW()` | +| `Unknown method X` | Function doesn't exist in LabKey SQL (check §6-§7) or is dialect-specific | +| `CONCAT function expects 2 arguments` | Use `\|\|` for 3+ values | +| `Syntax error near 'Count'` (or other keyword) | Reserved word used as alias — double-quote it | +| `HAVING requires an aggregate in the SELECT list` | Add the aggregate to the SELECT list or add GROUP BY | +| `Duplicate column 'x'` | Two select items produce the same name — alias one | +| `Ambiguous field: x` | Column exists in multiple FROM tables — qualify it | +| `VALUES expression requires an alias` | Add `AS t` after `(VALUES ...)` | +| `PIVOT queries must include a GROUP BY clause` | See §10 | +| `Could not find pivot column in group by list, expression must match exactly` | GROUP BY entry must textually match the PIVOT BY column's expression | +| `Parameter type is not supported: DATE` | Use TIMESTAMP (§11) | +| `The underlying database does not support nested ORDER BY unless LIMIT...` (warning) | ORDER BY is being dropped — add `LIMIT` or sort via the API | +| `ExecutingSelector; bad SQL grammar []` | Runtime (database-level) failure with no detail — usually a type mismatch (add CASTs) or a dialect-specific function limit (e.g. `timestampdiff` YEAR on PostgreSQL, `sum` over text) | diff --git a/query/src/org/labkey/query/sql/QMethodCall.java b/query/src/org/labkey/query/sql/QMethodCall.java index 1de51edf65d..2aff39739d8 100644 --- a/query/src/org/labkey/query/sql/QMethodCall.java +++ b/query/src/org/labkey/query/sql/QMethodCall.java @@ -139,7 +139,9 @@ public QueryParseException fieldCheck(QNode parent, SqlDialect d) { if (getMethod(d) == null) { - return new QueryParseException("Unknown method " + getField().getName(), null, getLine(), getColumn()); + String name = getField().getName(); + String hint = SyntaxHints.forUnknownMethod(name, d); + return new QueryParseException("Unknown method " + name + (null == hint ? "" : ". " + hint), null, getLine(), getColumn()); } return null; } diff --git a/query/src/org/labkey/query/sql/SqlBase.g b/query/src/org/labkey/query/sql/SqlBase.g index 72c652d9de7..349d8aab808 100644 --- a/query/src/org/labkey/query/sql/SqlBase.g +++ b/query/src/org/labkey/query/sql/SqlBase.g @@ -57,9 +57,6 @@ tokens @lexer::header { package org.labkey.query.sql.antlr; - - import org.apache.logging.log4j.Logger; - import org.apache.logging.log4j.LogManager; } @@ -134,17 +131,14 @@ tokens @lexer::members { - Logger _log = LogManager.getLogger(org.labkey.query.sql.SqlParser.class); - protected void setPossibleID(boolean possibleID) { } - @Override - public void emitErrorMessage(String msg) - { - _log.debug(msg); - } + // NOTE: lexer errors are reported via reportError(), which SqlParser._SqlLexer overrides to collect + // them as parse errors. Always use _SqlLexer rather than instantiating SqlBaseLexer directly -- the + // default ANTLR emitErrorMessage() prints unmatchable input to System.err and then drops it, letting + // the remaining characters re-lex into a different, valid-looking query. } diff --git a/query/src/org/labkey/query/sql/SqlParser.java b/query/src/org/labkey/query/sql/SqlParser.java index 62d84726a6d..9e137673bde 100644 --- a/query/src/org/labkey/query/sql/SqlParser.java +++ b/query/src/org/labkey/query/sql/SqlParser.java @@ -17,8 +17,10 @@ package org.labkey.query.sql; import org.antlr.runtime.ANTLRStringStream; +import org.antlr.runtime.CharStream; import org.antlr.runtime.CommonToken; import org.antlr.runtime.CommonTokenStream; +import org.antlr.runtime.MismatchedTokenException; import org.antlr.runtime.MissingTokenException; import org.antlr.runtime.ParserRuleReturnScope; import org.antlr.runtime.RecognitionException; @@ -35,6 +37,7 @@ import org.junit.Test; import org.labkey.api.collections.CaseInsensitiveHashSet; import org.labkey.api.data.Container; +import org.labkey.api.data.CoreSchema; import org.labkey.api.data.JdbcType; import org.labkey.api.data.dialect.SqlDialect; import org.labkey.api.module.Module; @@ -433,9 +436,10 @@ else if (null != warnings) errors.add(new QueryParseException("This does not look like a WITH, SELECT or UNION query", null, 0, 0)); } + CommonTokenStream tokens = parser.getTokenStream() instanceof CommonTokenStream cts ? cts : null; for (Throwable e : _parseErrors) { - errors.add(wrapParseException(e)); + errors.add(wrapParseException(e, tokens)); } if (null != _root) @@ -517,9 +521,10 @@ public QExpr parseExpr(String str, boolean constExpression, List"; // lexer error at end of input (e.g. unterminated string literal) + else if (re.c > 0) + near = String.valueOf((char)re.c); // lexer error: no token, but we know the offending character if (re instanceof MissingTokenException mte) { if (null != mte.inserted) missing = tokenName(((CommonToken)mte.inserted).getType()); } + else if (re instanceof MismatchedTokenException mte && null == re.token && mte.expecting > 0) + { + missing = String.valueOf((char)mte.expecting); // lexer error: expected character + } if (null != near) message = "Syntax error near '" + near + "'"; @@ -784,6 +803,11 @@ static String formatRecognitionException(RecognitionException re) message = "Syntax error"; if (null != missing) message += ", expected '" + missing + "'"; + + // append a targeted hint when the failure looks like a recognizable unsupported construct + String hint = SyntaxHints.forSyntaxError(re, tokens); + if (null != hint) + message += ". " + hint; return message; } @@ -1159,7 +1183,10 @@ else if (divisorType== SqlBaseParser.NUM_DOUBLE || divisorType== SqlBaseParser.N catch (IllegalArgumentException x) { if (failOnUnrecognizedMethodName) - _parseErrors.add(new QueryParseException("Unknown method " + name, null, id.getLine(), id.getColumn())); + { + String hint = SyntaxHints.forUnknownMethod(name, _dialect); + _parseErrors.add(new QueryParseException("Unknown method " + name + (null == hint ? "" : ". " + hint), null, id.getLine(), id.getColumn())); + } } } break; @@ -1727,6 +1754,30 @@ public int LA(int i) } + /** + * The default ANTLR lexer error handling prints unmatchable input to System.err and then drops it, + * letting the remaining characters re-lex into a different, valid-looking query (e.g. + * "{d'2001-02-03'}" -- missing the space after "{d" -- evaluated as 2001-02-03 = 1996 and swallowed + * the rest of the statement). Collect lexer errors so they surface as parse errors instead. + */ + private static class _SqlLexer extends SqlBaseLexer + { + private final ArrayList _errors; + + _SqlLexer(CharStream input, ArrayList errors) + { + super(input); + _errors = errors; + } + + @Override + public void reportError(RecognitionException e) + { + _errors.add(e); + } + } + + private static class _SqlParser extends SqlBaseParser implements AutoCloseable { ArrayList _errors; @@ -1748,7 +1799,7 @@ public _SqlParser(String str, ArrayList errors) public void reset(String str, ArrayList errors) { _errors = errors; - setTokenStream(new CommonTokenStream(new SqlBaseLexer(new CaseInsensitiveStringStream(str)))); + setTokenStream(new CommonTokenStream(new _SqlLexer(new CaseInsensitiveStringStream(str), errors))); } @Override @@ -2062,6 +2113,9 @@ class delete elements fetch indices insert into limit new set update versioned b "SELECT TIMESTAMPDIFF('SQL_TSI_Second',a,b), TIMESTAMPDIFF('Second',a,b), TIMESTAMPDIFF('SQL_TSI_Day',a,b), TIMESTAMPDIFF('Day',a,b) FROM R", "SELECT TIMESTAMPADD(SQL_TSI_SECOND,1,b), TIMESTAMPADD(SECOND,1,b), TIMESTAMPADD('SQL_TSI_DAY',1,b), TIMESTAMPADD('DAY',1,b) FROM R", + // date/timestamp literals (JDBC escape syntax; note the space after {d and {ts is required) + "SELECT {d '2001-02-03'} AS d, {ts '2001-02-03 04:05:06'} AS ts FROM R", + "SELECT (SELECT value FROM S WHERE S.x=R.x) AS V FROM R", "SELECT R.value AS V FROM R WHERE R.y > (SELECT MAX(S.y) FROM S WHERE S.x=R.x)", "SELECT R.value, T.a, T.b FROM R INNER JOIN (SELECT S.a, S.b FROM S) T ON R.z=T.z", @@ -2129,6 +2183,15 @@ class delete elements fetch indices insert into limit new set update versioned b "SELECT a, GROUP_CONCAT(b, '%$', 'STUPID') FROM R GROUP BY a", "SELECT a, GROUP_CONCAT() FROM R GROUP BY a", + // lexer errors must be reported, not silently dropped (see _SqlLexer) + // missing space after {d used to evaluate as arithmetic (2001-02-03 = 1996) and swallow the rest of the statement + "SELECT {d'2001-02-03'} AS d FROM R", + "SELECT {ts'2001-02-03 04:05:06'} AS ts FROM R", + // unmatchable character used to be dropped, silently parsing as "SELECT a b FROM R" + "SELECT a # b FROM R", + // unterminated string literal + "SELECT a FROM R WHERE a = 'unterminated", + "BROKEN", // empty select list @@ -2143,6 +2206,37 @@ class delete elements fetch indices insert into limit new set update versioned b "SELECT * FROM (WITH peeps AS (SELECT * FROM study.participant) SELECT * FROM peeps)" }; + // unsupported standard-SQL constructs that should fail with a targeted hint (see SyntaxHints): + // sql -> expected substring of the error message + static List> hintSql = Arrays.asList( + new Pair<>("SELECT a FROM R LIMIT 5 OFFSET 10", "OFFSET is not supported"), + new Pair<>("SELECT a FROM R ORDER BY a FETCH FIRST 5 ROWS ONLY", "FETCH FIRST is not supported"), + new Pair<>("SELECT ROW_NUMBER() OVER (ORDER BY a) FROM R", "Window functions"), + new Pair<>("SELECT SUM(x) OVER (PARTITION BY a) FROM R", "Window functions"), + new Pair<>("SELECT COUNT(*) FILTER (WHERE a > 0) FROM R", "FILTER is not supported"), + new Pair<>("SELECT a FROM R WHERE a ILIKE 'x%'", "ILIKE is not supported"), + new Pair<>("SELECT a FROM R JOIN S USING (x)", "USING is not supported"), + new Pair<>("SELECT a FROM R ORDER BY a NULLS LAST", "NULLS FIRST/LAST is not supported"), + new Pair<>("SELECT a::INTEGER FROM R", "CAST(expr AS TYPE)"), + new Pair<>("SELECT SUM(DISTINCT a) FROM R", "DISTINCT is only supported inside COUNT()"), + new Pair<>("SELECT a FROM R WHERE a IS DISTINCT FROM b", "is_distinct_from"), + new Pair<>("SELECT a FROM R WHERE a IS NOT DISTINCT FROM b", "is_distinct_from"), + new Pair<>("SELECT EXTRACT(YEAR FROM d) FROM R", "EXTRACT is not supported"), + new Pair<>("SELECT d + INTERVAL '1 day' FROM R", "INTERVAL literals are not supported"), + new Pair<>("SELECT TOP 10 a FROM R", "TOP is not supported") + ); + + // unrecognized method names that should fail with a suggested replacement (see SyntaxHints) + static List> methodHintSql = Arrays.asList( + new Pair<>("SELECT POSITION('a' IN b) FROM R", "LOCATE"), + new Pair<>("SELECT DATEDIFF('day', a, b) FROM R", "TIMESTAMPDIFF"), + new Pair<>("SELECT ISNULL(a, b) FROM R", "IFNULL"), + new Pair<>("SELECT DAY(a) FROM R", "DAYOFMONTH"), + new Pair<>("SELECT CURRENT_DATE() FROM R", "CURDATE"), + new Pair<>("SELECT STRING_AGG(a, ',') FROM R", "GROUP_CONCAT"), + new Pair<>("SELECT TRIM(a) FROM R", "LTRIM(RTRIM") + ); + @SuppressWarnings("JUnitMalformedDeclaration") public static class SqlParserTestCase extends Assert { @@ -2357,6 +2451,46 @@ public void testSql() _log.trace("SqlParser.testSql(): {}", DateUtil.formatDuration(end - start)); } + @Test + public void testSyntaxHints() + { + for (Pair test : hintSql) + { + List errors = new ArrayList<>(); + new SqlParser().parseQuery(test.first, errors, null); + assertFalse("expected a parse error: " + test.first, errors.isEmpty()); + assertTrue("expected error containing <<" + test.second + ">> for: " + test.first + "\nfound: " + errors.getFirst().getMessage(), + errors.stream().anyMatch(e -> StringUtils.contains(e.getMessage(), test.second))); + } + } + + @Test + public void testUnknownMethodHints() + { + for (Pair test : methodHintSql) + { + List errors = new ArrayList<>(); + new SqlParser().setFailOnUnrecognizedMethodName(true).parseQuery(test.first, errors, null); + assertFalse("expected a parse error: " + test.first, errors.isEmpty()); + assertTrue("expected error containing <<" + test.second + ">> for: " + test.first + "\nfound: " + errors.getFirst().getMessage(), + errors.stream().anyMatch(e -> StringUtils.contains(e.getMessage(), test.second))); + } + } + + @Test + public void testDialectMethodHints() + { + // the suggestion should be appropriate for the current dialect and never name a database product + SqlDialect d = CoreSchema.getInstance().getSqlDialect(); + List errors = new ArrayList<>(); + new SqlParser(d, null).setFailOnUnrecognizedMethodName(true).parseQuery("SELECT TRIM(a) FROM R", errors, null); + assertFalse(errors.isEmpty()); + String message = errors.getFirst().getMessage(); + assertTrue(message, StringUtils.contains(message, "LTRIM(RTRIM")); + assertEquals(message, d.isPostgreSQL(), StringUtils.contains(message, "btrim")); + assertFalse(message, StringUtils.containsIgnoreCase(message, "postgres")); + } + @Test public void testPrecedence() throws SQLException { diff --git a/query/src/org/labkey/query/sql/SyntaxHints.java b/query/src/org/labkey/query/sql/SyntaxHints.java new file mode 100644 index 00000000000..aa473f34f3a --- /dev/null +++ b/query/src/org/labkey/query/sql/SyntaxHints.java @@ -0,0 +1,150 @@ +/* + * Copyright (c) 2026 LabKey Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.labkey.query.sql; + +import org.antlr.runtime.CommonTokenStream; +import org.antlr.runtime.RecognitionException; +import org.antlr.runtime.Token; +import org.jetbrains.annotations.Nullable; +import org.labkey.api.data.dialect.SqlDialect; + +import java.util.Map; + +/** + * Suggestions for common standard-SQL constructs that LabKey SQL does not support. Authors (and AI assistants) + * regularly reach for window functions, OFFSET, EXTRACT, ILIKE, etc.; the generic "Syntax error near '...'" gives + * them no way to converge on working LabKey SQL, so we append a targeted hint when the failure looks recognizable. + * + * These hints are consulted ONLY after a parse error has already occurred, keyed off the token ANTLR blames (with a + * little look-behind for constructs where the blamed token is generic). None of the trigger words are reserved in + * LabKey SQL -- most are legal identifiers -- so this class must never influence what parses; message text only. + */ +final class SyntaxHints +{ + private SyntaxHints() + { + } + + @Nullable + static String forSyntaxError(RecognitionException re, @Nullable CommonTokenStream tokens) + { + if (null == re.token || null == re.token.getText()) + return null; + String near = re.token.getText().toLowerCase(); + String prev1 = previous(tokens, re.token, 1); + String prev2 = previous(tokens, re.token, 2); + String prev3 = previous(tokens, re.token, 3); + + switch (near) + { + case "offset": + return "OFFSET is not supported. Use LIMIT n; apply paging via the client API (maxRows/offset)."; + case "fetch": + return "FETCH FIRST is not supported. Use LIMIT n."; + case "ilike": + return "ILIKE is not supported. Use LOWER(x) LIKE LOWER(pattern)."; + case "using": + return "JOIN ... USING is not supported. Use JOIN ... ON a.col = b.col."; + case "nulls": + return "NULLS FIRST/LAST is not supported. Try ORDER BY x IS NULL, x."; + case ":": + return "The '::' cast syntax is not supported. Use CAST(expr AS TYPE)."; + case "(": + // "MAX(a) OVER (...)" parses OVER as a column alias, so the '(' gets the blame + if ("over".equals(prev1)) + return "Window functions (OVER) are not supported in LabKey SQL."; + if ("filter".equals(prev1)) + return "FILTER is not supported. Use an aggregate over CASE: SUM(CASE WHEN condition THEN 1 ELSE 0 END)."; + // "JOIN S USING (x)" parses USING as the table alias, so the '(' gets the blame + if ("using".equals(prev1)) + return "JOIN ... USING is not supported. Use JOIN ... ON a.col = b.col."; + return null; + case "distinct": + if ("is".equals(prev1) || "not".equals(prev1)) + return "IS [NOT] DISTINCT FROM is not supported. Use is_distinct_from(a, b) or is_not_distinct_from(a, b)."; + if ("(".equals(prev1)) + return "DISTINCT is only supported inside COUNT() and GROUP_CONCAT()."; + return null; + case "from": + // "EXTRACT(YEAR FROM d)" parses as a method call, so the FROM gets the blame + if ("(".equals(prev2) && "extract".equals(prev3)) + return "EXTRACT is not supported. Use YEAR(), MONTH(), DAYOFMONTH(), HOUR(), etc."; + return null; + default: + if (near.startsWith("'") && "interval".equals(prev1)) + return "INTERVAL literals are not supported. Use TIMESTAMPADD('SQL_TSI_DAY', n, ts) and TIMESTAMPDIFF()."; + // "SELECT TOP 10 a FROM R" parses TOP as an expression, so the blame lands on a later token + if (("top".equals(prev1) && "select".equals(prev2)) || ("top".equals(prev2) && "select".equals(prev3))) + return "TOP is not supported. Use LIMIT n at the end of the statement."; + return null; + } + } + + @Nullable + private static String previous(@Nullable CommonTokenStream tokens, Token t, int back) + { + if (null == tokens) + return null; + int i = t.getTokenIndex(); + if (i < back || i >= tokens.size()) + return null; + Token p = tokens.get(i - back); + return null == p || null == p.getText() ? null : p.getText().toLowerCase(); + } + + // Suggestions for unrecognized method names, keyed by lower-cased name. These entries are valid on both + // databases; dialect-specific suggestions live in forUnknownMethod(). Some entries (len, charindex, instr) + // are dialect-specific methods that resolve on one database and land here on the other. + private static final Map methodHints = Map.ofEntries( + Map.entry("position", "Use LOCATE(substring, string[, start])."), + Map.entry("extract", "Use YEAR(), MONTH(), DAYOFMONTH(), HOUR(), etc."), + Map.entry("string_agg", "Use GROUP_CONCAT([DISTINCT] expr[, separator])."), + Map.entry("nvl", "Use COALESCE(a, b) or IFNULL(a, b)."), + Map.entry("isnull", "Use IFNULL(a, b) or COALESCE(a, b)."), + Map.entry("iif", "Use CASE WHEN condition THEN a ELSE b END."), + Map.entry("if", "Use CASE WHEN condition THEN a ELSE b END."), + Map.entry("datediff", "Use TIMESTAMPDIFF('SQL_TSI_DAY', ts1, ts2) or AGE()/age_in_days()."), + Map.entry("dateadd", "Use TIMESTAMPADD('SQL_TSI_DAY', n, ts)."), + Map.entry("date_part", "Use YEAR(), MONTH(), DAYOFMONTH(), HOUR(), etc."), + Map.entry("day", "Use DAYOFMONTH(date)."), + Map.entry("len", "Use LENGTH(string)."), + Map.entry("instr", "Use LOCATE(substring, string)."), + Map.entry("charindex", "Use LOCATE(substring, string)."), + Map.entry("getdate", "Use NOW()."), + Map.entry("sysdate", "Use NOW()."), + Map.entry("current_date", "Use CURDATE()."), + Map.entry("current_timestamp", "Use NOW()."), + Map.entry("current_time", "Use CURTIME().") + ); + + /** + * The suggestion should simply be appropriate for the current dialect -- never name a database product. + * A null dialect (expression parsing, tests) gets the portable suggestion. + */ + @Nullable + static String forUnknownMethod(String name, @Nullable SqlDialect dialect) + { + boolean pg = null != dialect && dialect.isPostgreSQL(); + return switch (name.toLowerCase()) + { + case "trim" -> pg ? "Use btrim(x) or LTRIM(RTRIM(x))." : "Use LTRIM(RTRIM(x))."; + case "substring_index" -> pg ? "Use split_part(string, delimiter, n)." : null; + case "regexp_like", "regexp_matches" -> pg ? "Use similar_to(x, pattern) or regexp_replace(x, pattern, replacement)." : "Use LIKE with wildcards."; + case "date_trunc" -> pg ? "Use CAST(ts AS DATE) for day granularity, or to_char(ts, format)." : "Use CAST(ts AS DATE) for day granularity."; + default -> methodHints.get(name.toLowerCase()); + }; + } +}