diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0a07e2b1..5e07f454 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -71,6 +71,10 @@ jobs: working-directory: web run: bun install --frozen-lockfile + - name: Test + working-directory: web/default + run: bun run test + - name: Typecheck working-directory: web/default run: bun run typecheck diff --git "a/MAX_API_Next\347\244\276\345\214\272\344\273\213\347\273\215.md" "b/MAX_API_Next\347\244\276\345\214\272\344\273\213\347\273\215.md" deleted file mode 100644 index 0b381193..00000000 --- "a/MAX_API_Next\347\244\276\345\214\272\344\273\213\347\273\215.md" +++ /dev/null @@ -1,237 +0,0 @@ -# MAX API Next:面向 AI Models 与 Agents 治理的新一代开源社区 - -当 AI 应用从 Demo 走向真实业务,问题很快就不再只是“能不能调用一个模型”。 - -越来越多的团队开始同时接入 OpenAI、Claude、Gemini、Azure、AWS、Vertex、Ollama,以及 DeepSeek、通义千问、智谱 GLM、Kimi、豆包、混元、文心、讯飞星火、MiniMax 等国内外模型与平台。与此同时,Agent 工作流开始接管更多复杂任务:它们会连续调用模型、工具、知识库、图像、视频、音频和搜索能力,也会产生更长链路、更高成本和更复杂的权限边界。 - -这意味着,AI 应用时代真正需要被系统化解决的问题,已经从“模型调用”扩展为“模型与 Agent 的持续治理”。 - -MAX API Next 社区正是在这个背景下成立的。**社区由来自科研机构和高校的 AGI 爱好者组织发起、维护和运营,是一个研究驱动的 AI Models 与 Agents 治理项目**,长期关注模型平台适配、AgentOps 工程实践、成本审计、治理边界和 AI 应用基础设施建设。 - - - -![在这里插入图片描述](https://i-blog.csdnimg.cn/direct/9222cf388dec4deb826923061623ddb3.jpeg) - -## MAX API Next 是什么? - -MAX API Next 是一个聚焦 **AI Models 与 Agents 治理** 的技术社区。 - -社区围绕 AI 应用落地后的基础设施问题,持续建设可复用的工程能力: - -- 如何统一接入不同模型平台与协议? -- 如何让应用和 Agent 使用稳定、可控的模型入口? -- 如何管理 API Key、用户、组织、模型范围和额度? -- 如何在多供应商之间做路由、降级、重试与成本控制? -- 如何让模型价格、任务价格和 Agent 调用成本更直观? -- 如何在私有化部署中保留审计、日志、权限与安全边界? -- 如何持续跟踪国产模型、国际平台和多模态接口的快速变化? - -我们希望 MAX API Next 成为一个面向开发者、研究者、团队和组织的 AI 治理社区:把上游模型服务、Agent 应用、用户体系、成本审计和运营控制连接在一起,让 AI 应用可以更稳定、更可控、更可持续地运行。 - -## 为什么现在需要 AI Models 与 Agents 治理? - -过去,很多 AI 应用只需要一个模型 Key、一段调用代码和一个简单的代理服务。 - -但今天的 AI 应用已经明显不同: - -- 模型供应商越来越多,接口格式和参数差异频繁变化。 -- 国产模型和平台迭代很快,价格、能力、上下文、限流策略不断调整。 -- Agent 不再是单次问答,而是长链路、多步骤、多工具、多模型组合调用。 -- 多模态能力快速进入生产环境,图像、视频、音频、embedding、rerank、实时语音都需要统一管理。 -- 成本核算从“按 token”扩展到“按模型、按任务、按质量、按时长、按阶段、按供应商规则”。 -- 企业、团队和社区服务需要更清晰的用户、令牌、权限、日志和审计边界。 - -当这些问题叠加在一起,一个统一的治理层就变得非常关键。 - -MAX API Next 社区希望推动的方向,就是把 AI Models 和 Agents 的接入、路由、计费、权限、审计和运营能力沉淀为基础设施,而不是让每个应用重复实现一遍。 - -## MAX API:AGI 应用服务基础设施 -![在这里插入图片描述](https://i-blog.csdnimg.cn/direct/7d87f76424c74e31b4100ff7c97a9815.jpeg) - -MAX API 是 MAX API Next 社区中的核心开源项目之一。 - -它可以理解为一个面向 AI Models 与 Agents 的 API 网关、运营中台和治理层,位于应用、Agent、用户、组织和上游模型服务商之间。 - -应用侧可以通过统一接口访问不同模型能力;管理侧可以统一配置渠道、模型、价格、分组、令牌、日志、路由和安全策略;运营侧可以观察消耗、成本、错误、任务状态和使用情况。 - -![在这里插入图片描述](https://i-blog.csdnimg.cn/direct/8f965eed98c74f358b0cb9f61e93c00c.png) - -更具体地说,MAX API 希望解决三类核心问题: - -1. **模型接入问题**:多模型、多平台、多协议、多模态能力如何统一接入。 -2. **AgentOps 问题**:Agent 的令牌、权限、成本、调用链路和异常如何治理。 -3. **运营治理问题**:渠道、价格、日志、审计、安全和私有化部署如何长期维护。 -## 核心能力一:统一 AI 模型入口 - -MAX API 支持将不同类型的模型接口纳入统一治理,包括但不限于: - -- OpenAI-compatible 接口 -- OpenAI Responses -- Claude Messages -- Gemini -- Azure OpenAI -- AWS Bedrock -- Vertex AI -- Ollama -- embeddings -- rerank -- images -- audio -- realtime -- video tasks -- model discovery - -对于应用开发者来说,这意味着应用侧不必为每一家供应商维护一套完整的接入逻辑。对于团队和组织来说,这意味着模型入口、鉴权、路由和成本可以被统一管理。 - - -## 核心能力二:持续适配国产模型与平台 - -MAX API 特别重视国内模型和平台生态的持续跟踪适配。 - -国内 AI 平台的模型更新、接口变化、价格调整和多模态能力发布节奏很快。如果每个团队都单独跟进,会产生大量重复工作。 - -MAX API 将这类变化沉淀到统一的渠道配置、模型映射、协议转换、价格规则和任务协议中,帮助开发者更快接入和切换不同模型平台。 - -这不仅包括文本模型,也包括图像、视频、音频、embedding、rerank、Agent 应用平台和工作流平台等更广泛的 AI 能力生态。 - - -![在这里插入图片描述](https://i-blog.csdnimg.cn/direct/9b12f6b5f3f54509aad84c7b02b39d2b.png) - -## 核心能力三:面向 Agent 的治理能力 - -Agent 应用的复杂度通常高于普通聊天应用。 - -一个 Agent 可能会在一次任务中连续调用多个模型、多种工具和多个外部服务。如果缺少治理层,就很难回答这些问题: - -- 这个 Agent 能调用哪些模型? -- 它是否应该使用高成本模型? -- 它的额度是否需要单独限制? -- 它的调用失败发生在哪个上游渠道? -- 它的成本应该归因到哪个用户、团队或业务线? -- 出现异常时,管理员能否快速定位问题? - -MAX API 围绕 AgentOps 提供令牌隔离、模型范围控制、额度限制、渠道路由、失败重试、日志记录、成本统计和审计边界等能力,让 Agent 应用在真实业务中更容易被运营和管理。 - -## 核心能力四:更直观的计费与成本审计 - -AI 成本正在变得越来越复杂。 - -传统的“倍率计费”已经很难覆盖真实场景。不同模型可能存在输入 token、输出 token、缓存命中、图像 token、音频 token、分阶段价格、固定任务价格、视频时长、清晰度、是否带音频、是否有参考图等多种计费因素。 - -MAX API 在计费方向持续优化,重点包括: - -- 支持更直观的价格表达式。 -- 支持按真实价格维护模型计费规则。 -- 支持分阶段计费 JSON 批量配置。 -- 支持视频等异步任务的 rate-card 规则。 -- 支持按用户、令牌、模型、渠道和分组统计用量。 -- 支持预扣费、失败退款和任务完成后的成本归因。 - -对于运营者来说,计费配置不应该是一堆难以理解的倍率参数,而应该尽量接近上游价格表本身,便于核对、更新和审计。 - -## 核心能力五:渠道能力矩阵与配置校验 - -多渠道治理的难点之一,是管理员很难直观看到某个渠道到底支持哪些能力,也很难在保存配置前发现错误。 - -MAX API 正在强化渠道配置体验,例如: - -- 展示渠道能力矩阵。 -- 标识 chat/completions、responses、embeddings、rerank、images、audio、video tasks 等能力。 -- 对 Base URL、API Key、模型列表、JSON 配置、Vertex 区域、Codex 凭证、视频任务路径占位符等常见问题进行提示。 -- 通过更清晰的表单和校验减少误配置。 - -这类能力看起来不像模型本身那么显眼,但对长期运营非常重要。因为真实系统里的很多故障,往往不是模型不可用,而是渠道配置错误、协议路径变化、价格规则遗漏或权限边界不清。 - - -![在这里插入图片描述](https://i-blog.csdnimg.cn/direct/44c30d84ccf9403bba732cc4c412433b.png) -## 核心能力六:通用视频任务协议 - -视频生成、图像生成、长任务推理等异步任务越来越常见。 - -这类接口通常不是一次请求直接返回结果,而是经历: - -1. 提交任务。 -2. 获得任务 ID。 -3. 轮询任务状态。 -4. 解析进度、错误和结果。 -5. 返回代理后的内容地址。 -6. 按任务参数计费。 - -MAX API 将这类能力抽象为可配置的视频任务协议,支持任务提交路径、查询路径、状态映射、进度路径、错误消息路径、结果路径和内容代理等配置。 - -这样,不同上游平台即使接口路径和返回结构不同,也可以通过统一的任务协议纳入治理。 - -## 核心能力七:私有化部署与安全边界 - -MAX API 支持自托管部署,适合需要自主掌控密钥、数据、日志和策略的团队。 - -在私有化场景中,系统可以承担统一入口、用户管理、令牌管理、分组权限、模型范围、渠道路由、请求日志、错误日志、使用统计和安全限制等职责。 - -对于涉及输入输出内容记录的场景,MAX API 也强调审计边界:这类能力应当放在管理员端、安全与限制相关配置中,并在具备合法依据、明确告知、权限隔离和数据安全措施的前提下谨慎启用。 - - -![在这里插入图片描述](https://i-blog.csdnimg.cn/direct/aee0bc322cbb48909f94739f1ef948c7.png) -## MAX API 适合谁? - -MAX API 更适合以下几类使用者: - -- 正在维护多个模型供应商接入的开发者。 -- 需要统一管理模型 Key、用户、分组和额度的团队。 -- 正在建设 Agent 应用、工作流应用或内部 AI 平台的组织。 -- 需要接入国产模型、国际模型和多模态能力的项目。 -- 需要对 AI 调用成本进行统计、审计和归因的运营团队。 -- 需要私有化部署 AI API 网关的研究机构、高校团队和企业团队。 - -如果你的应用只调用一个固定模型,并且没有用户、权限、成本和路由管理需求,那么一个简单的 SDK 可能就够了。 - -但如果你正在面对多模型、多供应商、多 Agent、多用户、多计费规则和多模态任务,MAX API 这类治理层就会变得越来越有价值。 - -## 社区希望一起建设什么? - -MAX API Next 社区希望和更多开发者一起建设 AI Models 与 Agents 治理生态。 - -我们关注的问题包括: - -- 新模型与新平台的持续适配。 -- 国产模型生态的接口、价格和能力跟踪。 -- AgentOps 的工程实践与治理边界。 -- 多模态任务协议抽象。 -- 更直观的模型计费与成本审计。 -- 私有化部署、日志审计、安全限制和合规提示。 -- 面向真实生产环境的稳定性、可观测性和运维体验。 - -AI 应用的发展速度很快,但真正决定它能否长期运行的,往往是那些不在演示视频里的基础能力:权限、成本、路由、审计、配置、日志、任务状态、失败处理和持续维护。 - -MAX API Next 社区希望把这些能力做好,并把它们开放给更多正在建设 AI 应用和 Agent 系统的人。 - -## 结语 - -AI Models 是能力源头,Agents 是应用形态,而治理层决定了这些能力能否稳定进入真实系统。 - -MAX API Next 社区将继续围绕 AI Models 与 Agents 治理,推进模型接入、国产平台适配、AgentOps、计费审计、多模态任务和私有化运营能力建设。 - -MAX API 是这个方向上的一个重要实践。它不是要替代模型厂商,也不是要替代 Agent 编排框架,而是希望成为应用、Agent、用户、组织和上游模型服务之间稳定、可控、可运营的基础设施层。 - -欢迎关注 MAX API Next 社区,一起建设面向 AI 应用时代的 Models 与 Agents 治理基础设施。 - ---- - -**社区访问** - -GitHub地址:https://github.com/MAX-API-Next -社群交流:QQ(950126533) | 微信群(搜索MAX-API) - -**关键词** - -AI API Gateway / AI Models Governance / AgentOps / OpenAI-compatible / 国产模型适配 / 多模型路由 / 成本审计 / rate-card / 私有化部署 / 多模态任务治理 - -**合规提示** - -在面向公众提供生成式 AI 服务、商业化运营或接入第三方上游模型服务时,请自行完成所在司法辖区要求的备案、授权、内容安全、实名、日志留存、税务、支付、隐私保护和用户协议等合规事项。 - -**相关文献/项目** - -| [One API](https://github.com/songquanpeng/one-api) | MIT 协议 | -| [New API](https://github.com/QuantumNous/new-api) | AGPLv3 协议 | -| [Midjourney-Proxy](https://github.com/novicezk/midjourney-proxy) | Apache-2.0 协议 | -| [Suno API](https://github.com/Suno-API/Suno-API) | MIT 协议 | \ No newline at end of file diff --git a/common/constants.go b/common/constants.go index 484fc820..0943455f 100644 --- a/common/constants.go +++ b/common/constants.go @@ -226,15 +226,15 @@ var ( GlobalWebRateLimitDuration int64 CriticalRateLimitEnable bool - CriticalRateLimitNum = 20 + CriticalRateLimitNum = 200 CriticalRateLimitDuration int64 = 20 * 60 - CriticalRouteRateLimitNum = 20 + CriticalRouteRateLimitNum = 200 CriticalRouteRateLimitDuration int64 = 20 * 60 // Password login is limited by account as well as the shared API bucket. // This prevents a shared NAT address from making unrelated users unable to log in. LoginRateLimitEnable bool - LoginRateLimitNum = 10 + LoginRateLimitNum = 100 LoginRateLimitDuration int64 = 15 * 60 UploadRateLimitNum = 10 @@ -245,7 +245,7 @@ var ( // Per-user search rate limit (applies after authentication, keyed by user ID) SearchRateLimitEnable = true - SearchRateLimitNum = 10 + SearchRateLimitNum = 100 SearchRateLimitDuration int64 = 60 ) @@ -277,8 +277,9 @@ const ( ) const ( - TopUpStatusPending = "pending" - TopUpStatusSuccess = "success" - TopUpStatusFailed = "failed" - TopUpStatusExpired = "expired" + TopUpStatusPending = "pending" + TopUpStatusSuccess = "success" + TopUpStatusFailed = "failed" + TopUpStatusExpired = "expired" + TopUpStatusPaidReconciliation = "paid_reconciliation" ) diff --git a/common/init.go b/common/init.go index 8720cdc1..240b7e6a 100644 --- a/common/init.go +++ b/common/init.go @@ -127,29 +127,32 @@ func InitEnv() { GeminiSafetySetting = GetEnvOrDefaultString("GEMINI_SAFETY_SETTING", "BLOCK_NONE") CohereSafetySetting = GetEnvOrDefaultString("COHERE_SAFETY_SETTING", "NONE") - // Initialize rate limit variables + initRateLimitEnv() + initConstantEnv() +} + +func initRateLimitEnv() { GlobalApiRateLimitEnable = GetEnvOrDefaultBool("GLOBAL_API_RATE_LIMIT_ENABLE", true) GlobalApiRateLimitNum = GetEnvOrDefault("GLOBAL_API_RATE_LIMIT", 720) GlobalApiRateLimitDuration = int64(GetEnvOrDefault("GLOBAL_API_RATE_LIMIT_DURATION", 180)) GlobalWebRateLimitEnable = GetEnvOrDefaultBool("GLOBAL_WEB_RATE_LIMIT_ENABLE", true) - GlobalWebRateLimitNum = GetEnvOrDefault("GLOBAL_WEB_RATE_LIMIT", 60) + GlobalWebRateLimitNum = GetEnvOrDefault("GLOBAL_WEB_RATE_LIMIT", 600) GlobalWebRateLimitDuration = int64(GetEnvOrDefault("GLOBAL_WEB_RATE_LIMIT_DURATION", 180)) CriticalRateLimitEnable = GetEnvOrDefaultBool("CRITICAL_RATE_LIMIT_ENABLE", true) - CriticalRateLimitNum = GetEnvOrDefault("CRITICAL_RATE_LIMIT", 20) + CriticalRateLimitNum = GetEnvOrDefault("CRITICAL_RATE_LIMIT", 200) CriticalRateLimitDuration = int64(GetEnvOrDefault("CRITICAL_RATE_LIMIT_DURATION", 20*60)) - CriticalRouteRateLimitNum = GetEnvOrDefault("CRITICAL_ROUTE_RATE_LIMIT", 20) + CriticalRouteRateLimitNum = GetEnvOrDefault("CRITICAL_ROUTE_RATE_LIMIT", 200) CriticalRouteRateLimitDuration = int64(GetEnvOrDefault("CRITICAL_ROUTE_RATE_LIMIT_DURATION", 20*60)) LoginRateLimitEnable = GetEnvOrDefaultBool("LOGIN_RATE_LIMIT_ENABLE", true) - LoginRateLimitNum = GetEnvOrDefault("LOGIN_RATE_LIMIT", 10) + LoginRateLimitNum = GetEnvOrDefault("LOGIN_RATE_LIMIT", 100) LoginRateLimitDuration = int64(GetEnvOrDefault("LOGIN_RATE_LIMIT_DURATION", 15*60)) SearchRateLimitEnable = GetEnvOrDefaultBool("SEARCH_RATE_LIMIT_ENABLE", true) - SearchRateLimitNum = GetEnvOrDefault("SEARCH_RATE_LIMIT", 10) + SearchRateLimitNum = GetEnvOrDefault("SEARCH_RATE_LIMIT", 100) SearchRateLimitDuration = int64(GetEnvOrDefault("SEARCH_RATE_LIMIT_DURATION", 60)) - initConstantEnv() } func initConstantEnv() { diff --git a/common/init_rate_limit_test.go b/common/init_rate_limit_test.go new file mode 100644 index 00000000..18965a95 --- /dev/null +++ b/common/init_rate_limit_test.go @@ -0,0 +1,52 @@ +package common + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestRateLimitEnvironmentDefaults(t *testing.T) { + envKeys := []string{ + "GLOBAL_API_RATE_LIMIT_ENABLE", + "GLOBAL_API_RATE_LIMIT", + "GLOBAL_API_RATE_LIMIT_DURATION", + "GLOBAL_WEB_RATE_LIMIT_ENABLE", + "GLOBAL_WEB_RATE_LIMIT", + "GLOBAL_WEB_RATE_LIMIT_DURATION", + "CRITICAL_RATE_LIMIT_ENABLE", + "CRITICAL_RATE_LIMIT", + "CRITICAL_RATE_LIMIT_DURATION", + "CRITICAL_ROUTE_RATE_LIMIT", + "CRITICAL_ROUTE_RATE_LIMIT_DURATION", + "LOGIN_RATE_LIMIT_ENABLE", + "LOGIN_RATE_LIMIT", + "LOGIN_RATE_LIMIT_DURATION", + "SEARCH_RATE_LIMIT_ENABLE", + "SEARCH_RATE_LIMIT", + "SEARCH_RATE_LIMIT_DURATION", + } + for _, key := range envKeys { + t.Setenv(key, "") + } + + initRateLimitEnv() + + require.True(t, GlobalApiRateLimitEnable) + require.Equal(t, 720, GlobalApiRateLimitNum) + require.EqualValues(t, 180, GlobalApiRateLimitDuration) + require.True(t, GlobalWebRateLimitEnable) + require.Equal(t, 600, GlobalWebRateLimitNum) + require.EqualValues(t, 180, GlobalWebRateLimitDuration) + require.True(t, CriticalRateLimitEnable) + require.Equal(t, 200, CriticalRateLimitNum) + require.EqualValues(t, 20*60, CriticalRateLimitDuration) + require.Equal(t, 200, CriticalRouteRateLimitNum) + require.EqualValues(t, 20*60, CriticalRouteRateLimitDuration) + require.True(t, LoginRateLimitEnable) + require.Equal(t, 100, LoginRateLimitNum) + require.EqualValues(t, 15*60, LoginRateLimitDuration) + require.True(t, SearchRateLimitEnable) + require.Equal(t, 100, SearchRateLimitNum) + require.EqualValues(t, 60, SearchRateLimitDuration) +} diff --git a/common/quota_math.go b/common/quota_math.go index 77dc17ea..31945dce 100644 --- a/common/quota_math.go +++ b/common/quota_math.go @@ -129,3 +129,9 @@ func QuotaFromDecimalChecked(d decimal.Decimal) (int, *QuotaClamp) { f, _ := d.Round(0).Float64() return saturateQuota(f, "QuotaFromDecimal") } + +// QuotaFromDecimalStrict rejects unrepresentable values instead of silently +// saturating them. Payment and settlement inputs must use this form. +func QuotaFromDecimalStrict(d decimal.Decimal) (int, error) { + return strictQuota(QuotaFromDecimalChecked(d)) +} diff --git a/common/system_monitor.go b/common/system_monitor.go index a3981d74..53eb5594 100644 --- a/common/system_monitor.go +++ b/common/system_monitor.go @@ -27,9 +27,35 @@ type SystemStatus struct { CPUUsage float64 MemoryUsage float64 DiskUsage float64 + CPUValid bool + MemoryValid bool + DiskValid bool + ObservedAt time.Time } var latestSystemStatus atomic.Value +var systemStatusClock = time.Now +var systemCPUUsageProbe = func() (float64, bool) { + percents, err := cpu.Percent(0, false) + if err != nil || len(percents) == 0 { + return 0, false + } + return percents[0], true +} +var systemMemoryUsageProbe = func() (float64, bool) { + memInfo, err := mem.VirtualMemory() + if err != nil { + return 0, false + } + return memInfo.UsedPercent, true +} +var systemDiskUsageProbe = func() (float64, bool) { + diskInfo := GetDiskSpaceInfo() + if diskInfo.Total == 0 { + return 0, false + } + return diskInfo.UsedPercent, true +} var systemMonitorState struct { sync.Mutex cancel context.CancelFunc @@ -106,26 +132,26 @@ func StopSystemMonitor(ctx context.Context) error { } func updateSystemStatus() { - var status SystemStatus + status := SystemStatus{ObservedAt: systemStatusClock()} // CPU // 注意:cpu.Percent(0, false) 返回自上次调用以来的 CPU 使用率 // 如果是第一次调用,可能会返回错误或不准确的值,但在循环中会逐渐正常 - percents, err := cpu.Percent(0, false) - if err == nil && len(percents) > 0 { - status.CPUUsage = percents[0] + if usage, valid := systemCPUUsageProbe(); valid { + status.CPUUsage = usage + status.CPUValid = true } // Memory - memInfo, err := mem.VirtualMemory() - if err == nil { - status.MemoryUsage = memInfo.UsedPercent + if usage, valid := systemMemoryUsageProbe(); valid { + status.MemoryUsage = usage + status.MemoryValid = true } // Disk - diskInfo := GetDiskSpaceInfo() - if diskInfo.Total > 0 { - status.DiskUsage = diskInfo.UsedPercent + if usage, valid := systemDiskUsageProbe(); valid { + status.DiskUsage = usage + status.DiskValid = true } latestSystemStatus.Store(status) diff --git a/common/system_monitor_test.go b/common/system_monitor_test.go index 3b4f69ed..4f93b6f5 100644 --- a/common/system_monitor_test.go +++ b/common/system_monitor_test.go @@ -16,3 +16,35 @@ func TestSystemMonitorCanStopAndRestart(t *testing.T) { cancel() } } + +func TestUpdateSystemStatusPreservesProbeValidity(t *testing.T) { + originalClock := systemStatusClock + originalCPUProbe := systemCPUUsageProbe + originalMemoryProbe := systemMemoryUsageProbe + originalDiskProbe := systemDiskUsageProbe + originalStatus := GetSystemStatus() + t.Cleanup(func() { + systemStatusClock = originalClock + systemCPUUsageProbe = originalCPUProbe + systemMemoryUsageProbe = originalMemoryProbe + systemDiskUsageProbe = originalDiskProbe + latestSystemStatus.Store(originalStatus) + }) + + observedAt := time.Unix(1000, 0) + systemStatusClock = func() time.Time { return observedAt } + systemCPUUsageProbe = func() (float64, bool) { return 0, false } + systemMemoryUsageProbe = func() (float64, bool) { return 72.5, true } + systemDiskUsageProbe = func() (float64, bool) { return 0, false } + + updateSystemStatus() + status := GetSystemStatus() + + require.Equal(t, observedAt, status.ObservedAt) + require.False(t, status.CPUValid) + require.Zero(t, status.CPUUsage) + require.True(t, status.MemoryValid) + require.Equal(t, 72.5, status.MemoryUsage) + require.False(t, status.DiskValid) + require.Zero(t, status.DiskUsage) +} diff --git a/controller/channel_performance.go b/controller/channel_performance.go new file mode 100644 index 00000000..d50926b8 --- /dev/null +++ b/controller/channel_performance.go @@ -0,0 +1,68 @@ +package controller + +import ( + "net/http" + "strings" + + "github.com/MAX-API-Next/MAX-API/service" + "github.com/gin-gonic/gin" +) + +// GetChannelPerformance exposes the read-only legacy-log projection used by +// the administrator Smart Operations Center. It never probes, reroutes, +// disables or mutates a channel. +func GetChannelPerformance(c *gin.Context) { + numericQuery, err := parsePerformanceNumericQuery(c) + if err != nil { + respondInvalidPerformanceQuery(c, err, service.ErrInvalidChannelPerformanceQuery, "failed to query channel performance") + return + } + channelID, err := parseIntQuery(c, "channel_id") + if err != nil { + respondInvalidPerformanceQuery(c, err, service.ErrInvalidChannelPerformanceQuery, "failed to query channel performance") + return + } + query := service.ChannelPerformanceQuery{ + StartAt: numericQuery.StartAt, + EndAt: numericQuery.EndAt, + Hours: numericQuery.Hours, + Limit: numericQuery.Limit, + ChannelID: channelID, + ModelName: strings.TrimSpace(c.Query("model")), + Group: strings.TrimSpace(c.Query("group")), + } + + result, err := service.GetChannelPerformance(c.Request.Context(), query) + if err != nil { + respondPerformanceError(c, err, service.ErrInvalidChannelPerformanceQuery, "failed to query channel performance") + return + } + + c.JSON(http.StatusOK, gin.H{ + "success": true, + "data": result, + }) +} + +// GetChannelPerformanceDetail exposes a channel-scoped 24-hour projection. +// It is queried only when an administrator opens the channel detail sheet. +func GetChannelPerformanceDetail(c *gin.Context) { + channelID, err := parseIntQuery(c, "channel_id") + if err != nil { + respondInvalidPerformanceQuery(c, err, service.ErrInvalidChannelPerformanceQuery, "failed to query channel performance detail") + return + } + result, err := service.GetChannelPerformanceDetail( + c.Request.Context(), + channelID, + ) + if err != nil { + respondPerformanceError(c, err, service.ErrInvalidChannelPerformanceQuery, "failed to query channel performance detail") + return + } + + c.JSON(http.StatusOK, gin.H{ + "success": true, + "data": result, + }) +} diff --git a/controller/model_performance.go b/controller/model_performance.go new file mode 100644 index 00000000..3bb9967c --- /dev/null +++ b/controller/model_performance.go @@ -0,0 +1,58 @@ +package controller + +import ( + "net/http" + "strings" + + "github.com/MAX-API-Next/MAX-API/service" + "github.com/gin-gonic/gin" +) + +// GetModelPerformance exposes the read-only legacy-log projection used by the +// administrator Smart Operations Center. It aggregates eligible production +// logs by model and never mutates, probes or reroutes a channel. +func GetModelPerformance(c *gin.Context) { + numericQuery, err := parsePerformanceNumericQuery(c) + if err != nil { + respondInvalidPerformanceQuery(c, err, service.ErrInvalidModelPerformanceQuery, "failed to query model performance") + return + } + query := service.ModelPerformanceQuery{ + StartAt: numericQuery.StartAt, + EndAt: numericQuery.EndAt, + Hours: numericQuery.Hours, + Limit: numericQuery.Limit, + ModelName: strings.TrimSpace(c.Query("model")), + Group: strings.TrimSpace(c.Query("group")), + } + + result, err := service.GetModelPerformance(c.Request.Context(), query) + if err != nil { + respondPerformanceError(c, err, service.ErrInvalidModelPerformanceQuery, "failed to query model performance") + return + } + + c.JSON(http.StatusOK, gin.H{ + "success": true, + "data": result, + }) +} + +// GetModelPerformanceDetail exposes the existing perf_metrics projection to +// administrators without coupling Smart Operations to public pricing-module +// visibility. The detail window is fixed at the latest 24 hours. +func GetModelPerformanceDetail(c *gin.Context) { + result, err := service.GetModelPerformanceDetail( + c.Request.Context(), + strings.TrimSpace(c.Query("model")), + ) + if err != nil { + respondPerformanceError(c, err, service.ErrInvalidModelPerformanceQuery, "failed to query model performance detail") + return + } + + c.JSON(http.StatusOK, gin.H{ + "success": true, + "data": result, + }) +} diff --git a/controller/perf_metrics.go b/controller/perf_metrics.go index 1873c57e..6aa7ef44 100644 --- a/controller/perf_metrics.go +++ b/controller/perf_metrics.go @@ -20,11 +20,7 @@ func GetPerfMetricsSummary(c *gin.Context) { } } - activeGroups := lo.Keys(ratio_setting.GetGroupRatioCopy()) - for _, route := range setting.GetAutoRoutes() { - activeGroups = append(activeGroups, route.Key) - } - result, err := perfmetrics.QuerySummaryAll(hours, activeGroups) + result, err := perfmetrics.QuerySummaryAll(hours, activePerformanceGroups()) if err != nil { c.JSON(http.StatusInternalServerError, gin.H{ "success": false, @@ -57,9 +53,10 @@ func GetPerfMetrics(c *gin.Context) { } result, err := perfmetrics.Query(perfmetrics.QueryParams{ - Model: modelName, - Group: c.Query("group"), - Hours: hours, + Model: modelName, + Group: c.Query("group"), + Hours: hours, + AllowedGroups: activePerformanceGroups(), }) if err != nil { c.JSON(http.StatusInternalServerError, gin.H{ @@ -77,6 +74,14 @@ func GetPerfMetrics(c *gin.Context) { }) } +func activePerformanceGroups() []string { + groups := lo.Keys(ratio_setting.GetGroupRatioCopy()) + for _, route := range setting.GetAutoRoutes() { + groups = append(groups, route.Key) + } + return groups +} + func filterActiveGroups(groups []perfmetrics.GroupResult) []perfmetrics.GroupResult { activeRatios := ratio_setting.GetGroupRatioCopy() return lo.Filter(groups, func(g perfmetrics.GroupResult, _ int) bool { diff --git a/controller/performance_error.go b/controller/performance_error.go new file mode 100644 index 00000000..74bb9f6b --- /dev/null +++ b/controller/performance_error.go @@ -0,0 +1,29 @@ +package controller + +import ( + "errors" + "fmt" + "net/http" + + "github.com/MAX-API-Next/MAX-API/logger" + "github.com/gin-gonic/gin" +) + +func respondInvalidPerformanceQuery(c *gin.Context, err error, invalidQueryError error, fallbackMessage string) { + respondPerformanceError(c, fmt.Errorf("%w: %v", invalidQueryError, err), invalidQueryError, fallbackMessage) +} + +func respondPerformanceError(c *gin.Context, err error, invalidQueryError error, fallbackMessage string) { + status := http.StatusInternalServerError + message := fallbackMessage + if errors.Is(err, invalidQueryError) { + status = http.StatusBadRequest + message = err.Error() + } else { + logger.LogError(c.Request.Context(), fallbackMessage+": "+err.Error()) + } + c.JSON(status, gin.H{ + "success": false, + "message": message, + }) +} diff --git a/controller/performance_query.go b/controller/performance_query.go new file mode 100644 index 00000000..5c188500 --- /dev/null +++ b/controller/performance_query.go @@ -0,0 +1,82 @@ +package controller + +import ( + "strconv" + + "github.com/gin-gonic/gin" +) + +type performanceNumericQuery struct { + StartAt int64 + EndAt int64 + Hours int + Limit int +} + +func parsePerformanceNumericQuery(c *gin.Context) (performanceNumericQuery, error) { + startAt, err := parseInt64Query(c, "start") + if err != nil { + return performanceNumericQuery{}, err + } + endAt, err := parseInt64Query(c, "end") + if err != nil { + return performanceNumericQuery{}, err + } + hours, err := parseIntQuery(c, "hours") + if err != nil { + return performanceNumericQuery{}, err + } + limit, err := parseIntQuery(c, "limit") + if err != nil { + return performanceNumericQuery{}, err + } + return performanceNumericQuery{ + StartAt: startAt, + EndAt: endAt, + Hours: hours, + Limit: limit, + }, nil +} + +func parseIntQuery(c *gin.Context, key string) (int, error) { + value, supplied := numericQueryValue(c, key) + if !supplied { + return 0, nil + } + parsed, err := strconv.Atoi(value) + if err != nil { + return 0, &numericQueryError{key: key} + } + return parsed, nil +} + +func parseInt64Query(c *gin.Context, key string) (int64, error) { + value, supplied := numericQueryValue(c, key) + if !supplied { + return 0, nil + } + parsed, err := strconv.ParseInt(value, 10, 64) + if err != nil { + return 0, &numericQueryError{key: key} + } + return parsed, nil +} + +type numericQueryError struct { + key string +} + +func (err *numericQueryError) Error() string { + return err.key + " must be a valid integer" +} + +func numericQueryValue(c *gin.Context, key string) (string, bool) { + values, supplied := c.Request.URL.Query()[key] + if !supplied { + return "", false + } + if len(values) == 0 { + return "", true + } + return values[0], true +} diff --git a/controller/performance_query_test.go b/controller/performance_query_test.go new file mode 100644 index 00000000..1be56bfe --- /dev/null +++ b/controller/performance_query_test.go @@ -0,0 +1,62 @@ +package controller + +import ( + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/gin-gonic/gin" + "github.com/stretchr/testify/require" +) + +func TestPerformanceHandlersRejectMalformedNumericQueryParameters(t *testing.T) { + gin.SetMode(gin.TestMode) + tests := []struct { + name string + path string + handler gin.HandlerFunc + key string + }{ + {name: "channel start", path: "/channel-performance?start=invalid", handler: GetChannelPerformance, key: "start"}, + {name: "channel end", path: "/channel-performance?end=invalid", handler: GetChannelPerformance, key: "end"}, + {name: "channel hours", path: "/channel-performance?hours=invalid", handler: GetChannelPerformance, key: "hours"}, + {name: "channel empty limit", path: "/channel-performance?limit=", handler: GetChannelPerformance, key: "limit"}, + {name: "channel id", path: "/channel-performance?channel_id=invalid", handler: GetChannelPerformance, key: "channel_id"}, + {name: "channel detail id", path: "/channel-performance/detail?channel_id=invalid", handler: GetChannelPerformanceDetail, key: "channel_id"}, + {name: "model start", path: "/model-performance?start=invalid", handler: GetModelPerformance, key: "start"}, + {name: "model end", path: "/model-performance?end=invalid", handler: GetModelPerformance, key: "end"}, + {name: "model hours", path: "/model-performance?hours=invalid", handler: GetModelPerformance, key: "hours"}, + {name: "model limit", path: "/model-performance?limit=invalid", handler: GetModelPerformance, key: "limit"}, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + recorder := httptest.NewRecorder() + ctx, _ := gin.CreateTestContext(recorder) + ctx.Request = httptest.NewRequest(http.MethodGet, test.path, nil) + + test.handler(ctx) + + require.Equal(t, http.StatusBadRequest, recorder.Code) + require.Contains(t, strings.ToLower(recorder.Body.String()), test.key) + }) + } +} + +func TestParsePerformanceNumericQueryPreservesZeroForAbsentParameters(t *testing.T) { + gin.SetMode(gin.TestMode) + ctx, _ := gin.CreateTestContext(httptest.NewRecorder()) + ctx.Request = httptest.NewRequest(http.MethodGet, "/performance", nil) + + query, err := parsePerformanceNumericQuery(ctx) + + require.NoError(t, err) + require.Zero(t, query.StartAt) + require.Zero(t, query.EndAt) + require.Zero(t, query.Hours) + require.Zero(t, query.Limit) + channelID, err := parseIntQuery(ctx, "channel_id") + require.NoError(t, err) + require.Zero(t, channelID) +} diff --git a/controller/smart_ops_alert.go b/controller/smart_ops_alert.go new file mode 100644 index 00000000..db14a395 --- /dev/null +++ b/controller/smart_ops_alert.go @@ -0,0 +1,17 @@ +package controller + +import ( + "net/http" + + "github.com/MAX-API-Next/MAX-API/service" + "github.com/gin-gonic/gin" +) + +// GetSmartOpsAlerts exposes the current in-process operational alerts to +// administrators. The endpoint is read-only and does not trigger remediation. +func GetSmartOpsAlerts(c *gin.Context) { + c.JSON(http.StatusOK, gin.H{ + "success": true, + "data": service.GetSmartOpsAlerts(), + }) +} diff --git a/controller/topup.go b/controller/topup.go index a204191d..db58899d 100644 --- a/controller/topup.go +++ b/controller/topup.go @@ -1,6 +1,7 @@ package controller import ( + "errors" "fmt" "net/http" "net/url" @@ -187,6 +188,38 @@ func getMinTopup() int64 { return int64(minTopup) } +func topUpCreditedQuota(amount int64) decimal.Decimal { + quota := decimal.NewFromInt(amount) + if operation_setting.GetQuotaDisplayType() == operation_setting.QuotaDisplayTypeTokens { + quotaPerUnit := decimal.NewFromFloat(common.QuotaPerUnit) + return decimal.NewFromInt(quota.Div(quotaPerUnit).IntPart()).Mul(quotaPerUnit) + } + return quota.Mul(decimal.NewFromFloat(common.QuotaPerUnit)) +} + +func rejectInvalidTopUpQuota(c *gin.Context, userId int, quota decimal.Decimal) bool { + credited, err := common.QuotaFromDecimalStrict(quota) + if err == nil && credited <= 0 { + err = model.ErrInvalidTopUpQuota + } + if err == nil { + err = model.ValidateTopUpQuotaCapacity(userId, int64(credited)) + } + if err != nil { + logger.LogError(c.Request.Context(), fmt.Sprintf("充值额度校验失败 user_id=%d error=%q", userId, err.Error())) + message := "充值额度校验失败,请稍后重试" + switch { + case errors.Is(err, model.ErrInvalidTopUpQuota): + message = "充值额度无效" + case errors.Is(err, model.ErrTopUpQuotaLimitExceeded): + message = "充值后额度将超过账户上限" + } + c.JSON(http.StatusOK, gin.H{"message": "error", "data": message}) + return true + } + return false +} + func RequestEpay(c *gin.Context) { var req EpayRequest err := c.ShouldBindJSON(&req) @@ -205,6 +238,9 @@ func RequestEpay(c *gin.Context) { c.JSON(http.StatusOK, gin.H{"message": "error", "data": "获取用户分组失败"}) return } + if rejectInvalidTopUpQuota(c, id, topUpCreditedQuota(req.Amount)) { + return + } payMoney := getPayMoney(req.Amount, group) if payMoney < 0.01 { c.JSON(http.StatusOK, gin.H{"message": "error", "data": "充值金额过低"}) @@ -382,6 +418,11 @@ func EpayNotify(c *gin.Context) { defer UnlockOrder(verifyInfo.ServiceTradeNo) if err := model.RechargeEpay(verifyInfo.ServiceTradeNo, verifyInfo.Type, c.ClientIP(), model.PaymentValidationFromMajorString(verifyInfo.Money, "", "", false)); err != nil { + if errors.Is(err, model.ErrTopUpNeedsReconciliation) { + logger.LogWarn(c.Request.Context(), fmt.Sprintf("易支付 已付款充值订单等待人工对账 trade_no=%s callback_type=%s client_ip=%s", verifyInfo.ServiceTradeNo, verifyInfo.Type, c.ClientIP())) + _, _ = c.Writer.Write([]byte("success")) + return + } logger.LogError(c.Request.Context(), fmt.Sprintf("易支付 充值处理失败 trade_no=%s callback_type=%s client_ip=%s error=%q", verifyInfo.ServiceTradeNo, verifyInfo.Type, c.ClientIP(), err.Error())) _, _ = c.Writer.Write([]byte("fail")) return diff --git a/controller/topup_creem.go b/controller/topup_creem.go index 6a76b900..5ab2af3d 100644 --- a/controller/topup_creem.go +++ b/controller/topup_creem.go @@ -17,6 +17,7 @@ import ( "time" "github.com/gin-gonic/gin" + "github.com/shopspring/decimal" "github.com/thanhpk/randstr" ) @@ -130,7 +131,14 @@ func (*CreemAdaptor) RequestPay(c *gin.Context, req *CreemPayRequest) { } id := c.GetInt("id") - user, _ := model.GetUserById(id, false) + user, err := model.GetUserById(id, false) + if err != nil || user == nil { + c.JSON(http.StatusOK, gin.H{"message": "error", "data": "用户不存在"}) + return + } + if rejectInvalidTopUpQuota(c, id, decimal.NewFromInt(int64(selectedProduct.Quota))) { + return + } // 生成唯一的订单引用ID reference := fmt.Sprintf("creem-api-ref-%d-%d-%s", user.Id, time.Now().UnixMilli(), randstr.String(4)) @@ -388,6 +396,11 @@ func handleCheckoutCompleted(c *gin.Context, event *CreemWebhookEvent) { } err = model.RechargeCreem(referenceId, customerEmail, customerName, c.ClientIP(), topUpValidation) if err != nil { + if errors.Is(err, model.ErrTopUpNeedsReconciliation) { + logger.LogWarn(c.Request.Context(), fmt.Sprintf("Creem 已付款充值订单等待人工对账 trade_no=%s creem_order_id=%s client_ip=%s", referenceId, event.Object.Order.Id, c.ClientIP())) + c.Status(http.StatusOK) + return + } logger.LogError(c.Request.Context(), fmt.Sprintf("Creem 充值处理失败 trade_no=%s creem_order_id=%s client_ip=%s error=%q", referenceId, event.Object.Order.Id, c.ClientIP(), err.Error())) c.AbortWithStatus(http.StatusInternalServerError) return diff --git a/controller/topup_quota_test.go b/controller/topup_quota_test.go new file mode 100644 index 00000000..96acb370 --- /dev/null +++ b/controller/topup_quota_test.go @@ -0,0 +1,46 @@ +package controller + +import ( + "fmt" + "net/http/httptest" + "strings" + "testing" + + "github.com/MAX-API-Next/MAX-API/common" + "github.com/MAX-API-Next/MAX-API/model" + "github.com/gin-gonic/gin" + "github.com/glebarez/sqlite" + "github.com/shopspring/decimal" + "github.com/stretchr/testify/require" + "gorm.io/gorm" +) + +func TestRejectInvalidTopUpQuotaDoesNotExposeDatabaseErrors(t *testing.T) { + oldDB := model.DB + db, err := gorm.Open(sqlite.Open(fmt.Sprintf("file:%s?mode=memory&cache=shared", strings.ReplaceAll(t.Name(), "/", "_"))), &gorm.Config{}) + require.NoError(t, err) + model.DB = db + t.Cleanup(func() { + model.DB = oldDB + }) + + sqlDB, err := db.DB() + require.NoError(t, err) + require.NoError(t, sqlDB.Close()) + + gin.SetMode(gin.TestMode) + recorder := httptest.NewRecorder() + c, _ := gin.CreateTestContext(recorder) + c.Request = httptest.NewRequest("POST", "/api/user/topup", nil) + + require.True(t, rejectInvalidTopUpQuota(c, 1, decimal.NewFromInt(1))) + + var response struct { + Message string `json:"message"` + Data string `json:"data"` + } + require.NoError(t, common.Unmarshal(recorder.Body.Bytes(), &response)) + require.Equal(t, "error", response.Message) + require.Equal(t, "充值额度校验失败,请稍后重试", response.Data) + require.NotContains(t, response.Data, "closed") +} diff --git a/controller/topup_stripe.go b/controller/topup_stripe.go index ef730e06..ed45246b 100644 --- a/controller/topup_stripe.go +++ b/controller/topup_stripe.go @@ -17,6 +17,7 @@ import ( "github.com/MAX-API-Next/MAX-API/setting/operation_setting" "github.com/gin-gonic/gin" + "github.com/shopspring/decimal" "github.com/stripe/stripe-go/v81" "github.com/stripe/stripe-go/v81/checkout/session" "github.com/stripe/stripe-go/v81/webhook" @@ -74,7 +75,6 @@ func (*StripeAdaptor) RequestPay(c *gin.Context, req *StripePayRequest) { c.JSON(http.StatusOK, gin.H{"message": "充值数量不能大于 10000", "data": 10}) return } - if req.SuccessURL != "" && common.ValidateRedirectURL(req.SuccessURL) != nil { c.JSON(http.StatusBadRequest, gin.H{"message": "支付成功重定向URL不在可信任域名列表中", "data": ""}) return @@ -86,8 +86,15 @@ func (*StripeAdaptor) RequestPay(c *gin.Context, req *StripePayRequest) { } id := c.GetInt("id") - user, _ := model.GetUserById(id, false) + user, err := model.GetUserById(id, false) + if err != nil || user == nil { + c.JSON(http.StatusOK, gin.H{"message": "error", "data": "用户不存在"}) + return + } chargedMoney := GetChargedAmount(float64(req.Amount), *user) + if rejectInvalidTopUpQuota(c, id, decimal.NewFromFloat(chargedMoney).Mul(decimal.NewFromFloat(common.QuotaPerUnit))) { + return + } reference := fmt.Sprintf("max-api-ref-%d-%d-%s", user.Id, time.Now().UnixMilli(), randstr.String(4)) referenceId := "ref_" + common.Sha1([]byte(reference)) @@ -235,7 +242,7 @@ func stripePaymentValidationFromEvent(event stripe.Event, allowDiscount bool) (m return model.PaymentValidationFromMinorUnits(amountMinor, event.GetObjectValue("currency"), "", allowDiscount), nil } -func isStripeOrderAlreadyCompleted(referenceId string) bool { +func isStripeOrderTerminallyHandled(referenceId string) bool { if order := model.GetSubscriptionOrderByTradeNo(referenceId); order != nil && order.PaymentProvider == model.PaymentProviderStripe && order.Status == common.TopUpStatusSuccess { @@ -244,7 +251,7 @@ func isStripeOrderAlreadyCompleted(referenceId string) bool { topUp, err := model.GetTopUpByTradeNoWithError(referenceId) return err == nil && topUp.PaymentProvider == model.PaymentProviderStripe && - topUp.Status == common.TopUpStatusSuccess + (topUp.Status == common.TopUpStatusSuccess || topUp.Status == common.TopUpStatusPaidReconciliation) } // sessionAsyncPaymentFailed marks orders as failed when delayed payment methods @@ -297,7 +304,7 @@ func fulfillOrder(ctx context.Context, event stripe.Event, referenceId string, c defer UnlockOrder(referenceId) validation, err := stripePaymentValidationFromEvent(event, setting.StripePromotionCodesEnabled) if err != nil { - if isStripeOrderAlreadyCompleted(referenceId) { + if isStripeOrderTerminallyHandled(referenceId) { logger.LogInfo(ctx, fmt.Sprintf("Stripe 订单已完成,确认缺少金额字段的重放事件 trade_no=%s event_type=%s client_ip=%s", referenceId, string(event.Type), callerIp)) return nil } @@ -346,6 +353,10 @@ func fulfillOrder(ctx context.Context, event stripe.Event, referenceId string, c topUpValidation := validation topUpValidation.ExpectedCurrency = "USD" if err := model.Recharge(referenceId, customerId, callerIp, topUpValidation); err != nil { + if errors.Is(err, model.ErrTopUpNeedsReconciliation) { + logger.LogWarn(ctx, fmt.Sprintf("Stripe 已付款充值订单等待人工对账 trade_no=%s client_ip=%s", referenceId, callerIp)) + return nil + } latest, latestErr := model.GetTopUpByTradeNoWithError(referenceId) if latestErr == nil && latest.Status == common.TopUpStatusSuccess { logger.LogInfo(ctx, fmt.Sprintf("Stripe 充值订单已由并发回调完成 trade_no=%s client_ip=%s", referenceId, callerIp)) diff --git a/controller/topup_stripe_test.go b/controller/topup_stripe_test.go index 0a56b8a4..1c401ce7 100644 --- a/controller/topup_stripe_test.go +++ b/controller/topup_stripe_test.go @@ -57,6 +57,8 @@ func completedStripeEvent(tradeNo string) stripe.Event { "customer": "cus_test", "status": "complete", "payment_status": "paid", + "amount_total": "100", + "currency": "usd", }}, } } @@ -104,3 +106,37 @@ func TestStripeSessionCompletedReturnsRetryableErrorWhenOrderLookupFails(t *test err = sessionCompleted(context.Background(), completedStripeEvent("stripe-database-unavailable"), "127.0.0.1") require.Error(t, err) } + +func TestStripeSessionCompletedAcknowledgesPaidOrderNeedingReconciliation(t *testing.T) { + db := setupStripeWebhookTestDB(t) + oldQuotaPerUnit := common.QuotaPerUnit + common.QuotaPerUnit = 500000 + t.Cleanup(func() { common.QuotaPerUnit = oldQuotaPerUnit }) + user := model.User{ + Id: 8104, + Username: "stripe-reconciliation-user", + Status: common.UserStatusEnabled, + Quota: int64(common.MaxQuota - 2), + } + require.NoError(t, db.Create(&user).Error) + topUp := model.TopUp{ + Id: 8104, + UserId: user.Id, + Money: 1, + TradeNo: "stripe-paid-reconciliation", + PaymentMethod: model.PaymentMethodStripe, + PaymentProvider: model.PaymentProviderStripe, + Status: common.TopUpStatusPending, + } + require.NoError(t, db.Create(&topUp).Error) + + require.NoError(t, sessionCompleted(context.Background(), completedStripeEvent(topUp.TradeNo), "127.0.0.1")) + require.NoError(t, sessionCompleted(context.Background(), completedStripeEvent(topUp.TradeNo), "127.0.0.1")) + + var stored model.TopUp + require.NoError(t, db.First(&stored, topUp.Id).Error) + require.Equal(t, common.TopUpStatusPaidReconciliation, stored.Status) + var storedUser model.User + require.NoError(t, db.First(&storedUser, user.Id).Error) + require.EqualValues(t, common.MaxQuota-2, storedUser.Quota) +} diff --git a/controller/topup_waffo.go b/controller/topup_waffo.go index 625fa865..d89155b1 100644 --- a/controller/topup_waffo.go +++ b/controller/topup_waffo.go @@ -16,6 +16,7 @@ import ( "github.com/MAX-API-Next/MAX-API/setting" "github.com/MAX-API-Next/MAX-API/setting/operation_setting" "github.com/gin-gonic/gin" + "github.com/shopspring/decimal" "github.com/thanhpk/randstr" waffo "github.com/waffo-com/waffo-go" "github.com/waffo-com/waffo-go/config" @@ -82,25 +83,48 @@ func formatWaffoAmount(amount float64, currency string) string { return fmt.Sprintf("%.2f", amount) } -// getWaffoPayMoney converts the user-facing amount to USD for Waffo payment. -// Waffo only accepts USD, so this function handles the conversion from different -// display types (USD/CNY/TOKENS) to the actual USD amount to charge. -func getWaffoPayMoney(amount float64, group string) float64 { - originalAmount := amount - if operation_setting.GetQuotaDisplayType() == operation_setting.QuotaDisplayTypeTokens { - amount = amount / common.QuotaPerUnit - } +type waffoTopUpQuote struct { + amount int64 + creditedQuota decimal.Decimal + payMoney float64 +} + +// buildWaffoTopUpQuote derives the persisted amount, credited quota, and +// payment price from one normalized value so settlement cannot credit more +// than the external order was priced for. +func buildWaffoTopUpQuote(requestedAmount int64, group string) waffoTopUpQuote { + amount := normalizeWaffoTopUpAmount(requestedAmount) + creditedQuota := decimal.NewFromInt(amount).Mul(decimal.NewFromFloat(common.QuotaPerUnit)) topupGroupRatio := common.GetTopupGroupRatio(group) if topupGroupRatio == 0 { topupGroupRatio = 1 } discount := 1.0 - if ds, ok := operation_setting.GetPaymentSetting().AmountDiscount[int(originalAmount)]; ok { + if ds, ok := operation_setting.GetPaymentSetting().AmountDiscount[int(requestedAmount)]; ok { if ds > 0 { discount = ds } } - return amount * setting.WaffoUnitPrice * topupGroupRatio * discount + payMoney := decimal.NewFromInt(amount). + Mul(decimal.NewFromFloat(setting.WaffoUnitPrice)). + Mul(decimal.NewFromFloat(topupGroupRatio)). + Mul(decimal.NewFromFloat(discount)). + InexactFloat64() + return waffoTopUpQuote{amount: amount, creditedQuota: creditedQuota, payMoney: payMoney} +} + +func normalizeWaffoTopUpAmount(amount int64) int64 { + if operation_setting.GetQuotaDisplayType() != operation_setting.QuotaDisplayTypeTokens { + return amount + } + + normalized := decimal.NewFromInt(amount). + Div(decimal.NewFromFloat(common.QuotaPerUnit)). + IntPart() + if normalized < 1 { + return 1 + } + return normalized } type WaffoPayRequest struct { @@ -130,7 +154,8 @@ func RequestWaffoAmount(c *gin.Context) { return } - payMoney := getWaffoPayMoney(float64(req.Amount), group) + quote := buildWaffoTopUpQuote(req.Amount, group) + payMoney := quote.payMoney if payMoney <= 0.01 { c.JSON(http.StatusOK, gin.H{"message": "error", "data": "充值金额过低"}) return @@ -196,8 +221,16 @@ func RequestWaffoPay(c *gin.Context) { } // resolvedPayMethodType/Name 为空时,Waffo 自动选择支付方式 - group, _ := model.GetUserGroup(id, true) - payMoney := getWaffoPayMoney(float64(req.Amount), group) + group, err := model.GetUserGroup(id, true) + if err != nil { + c.JSON(http.StatusOK, gin.H{"message": "error", "data": "获取用户分组失败"}) + return + } + quote := buildWaffoTopUpQuote(req.Amount, group) + if rejectInvalidTopUpQuota(c, id, quote.creditedQuota) { + return + } + payMoney := quote.payMoney if payMoney < 0.01 { c.JSON(http.StatusOK, gin.H{"message": "error", "data": "充值金额过低"}) return @@ -208,13 +241,7 @@ func RequestWaffoPay(c *gin.Context) { paymentRequestId := merchantOrderId // Token 模式下归一化 Amount(存等价美元/CNY 数量,避免 RechargeWaffo 双重放大) - amount := req.Amount - if operation_setting.GetQuotaDisplayType() == operation_setting.QuotaDisplayTypeTokens { - amount = int64(float64(req.Amount) / common.QuotaPerUnit) - if amount < 1 { - amount = 1 - } - } + amount := quote.amount // 创建本地订单 topUp := &model.TopUp{ @@ -403,6 +430,11 @@ func handleWaffoPayment(c *gin.Context, wh *core.WebhookHandler, result *core.Pa if err := model.RechargeWaffo(merchantOrderId, c.ClientIP(), model.PaymentValidationFromMajorString(getWaffoPaidAmount(result), result.OrderCurrency, getWaffoCurrency(), false)); err != nil { + if errors.Is(err, model.ErrTopUpNeedsReconciliation) { + logger.LogWarn(c.Request.Context(), fmt.Sprintf("Waffo 已付款充值订单等待人工对账 trade_no=%s client_ip=%s", merchantOrderId, c.ClientIP())) + sendWaffoWebhookResponse(c, wh, true, "") + return + } logger.LogError(c.Request.Context(), fmt.Sprintf("Waffo 充值处理失败 trade_no=%s client_ip=%s error=%q", merchantOrderId, c.ClientIP(), err.Error())) sendWaffoWebhookResponse(c, wh, false, err.Error()) return diff --git a/controller/topup_waffo_pancake.go b/controller/topup_waffo_pancake.go index 6b9e7fb8..e08066c1 100644 --- a/controller/topup_waffo_pancake.go +++ b/controller/topup_waffo_pancake.go @@ -1,6 +1,7 @@ package controller import ( + "errors" "fmt" "io" "net/http" @@ -40,8 +41,12 @@ func RequestWaffoPancakeAmount(c *gin.Context) { c.JSON(http.StatusOK, gin.H{"message": "error", "data": "获取用户分组失败"}) return } + quote := buildWaffoPancakeTopUpQuote(req.Amount, group) + if rejectInvalidTopUpQuota(c, id, quote.creditedQuota) { + return + } - payMoney := getWaffoPancakePayMoney(req.Amount, group) + payMoney := quote.payMoney if payMoney <= 0.01 { c.JSON(http.StatusOK, gin.H{"message": "error", "data": "充值金额过低"}) return @@ -50,11 +55,15 @@ func RequestWaffoPancakeAmount(c *gin.Context) { c.JSON(http.StatusOK, gin.H{"message": "success", "data": fmt.Sprintf("%.2f", payMoney)}) } -func getWaffoPancakePayMoney(amount int64, group string) float64 { - dAmount := decimal.NewFromInt(amount) - if operation_setting.GetQuotaDisplayType() == operation_setting.QuotaDisplayTypeTokens { - dAmount = dAmount.Div(decimal.NewFromFloat(common.QuotaPerUnit)) - } +type waffoPancakeTopUpQuote struct { + amount int64 + creditedQuota decimal.Decimal + payMoney float64 +} + +func buildWaffoPancakeTopUpQuote(requestedAmount int64, group string) waffoPancakeTopUpQuote { + amount := normalizeWaffoPancakeTopUpAmount(requestedAmount) + creditedQuota := decimal.NewFromInt(amount).Mul(decimal.NewFromFloat(common.QuotaPerUnit)) topupGroupRatio := common.GetTopupGroupRatio(group) if topupGroupRatio == 0 { @@ -62,16 +71,17 @@ func getWaffoPancakePayMoney(amount int64, group string) float64 { } discount := 1.0 - if ds, ok := operation_setting.GetPaymentSetting().AmountDiscount[int(amount)]; ok && ds > 0 { + if ds, ok := operation_setting.GetPaymentSetting().AmountDiscount[int(requestedAmount)]; ok && ds > 0 { discount = ds } - payMoney := dAmount. + payMoney := decimal.NewFromInt(amount). Mul(decimal.NewFromFloat(setting.WaffoPancakeUnitPrice)). Mul(decimal.NewFromFloat(topupGroupRatio)). - Mul(decimal.NewFromFloat(discount)) + Mul(decimal.NewFromFloat(discount)). + InexactFloat64() - return payMoney.InexactFloat64() + return waffoPancakeTopUpQuote{amount: amount, creditedQuota: creditedQuota, payMoney: payMoney} } func normalizeWaffoPancakeTopUpAmount(amount int64) int64 { @@ -373,8 +383,12 @@ func RequestWaffoPancakePay(c *gin.Context) { c.JSON(http.StatusOK, gin.H{"message": "error", "data": "获取用户分组失败"}) return } + quote := buildWaffoPancakeTopUpQuote(req.Amount, group) + if rejectInvalidTopUpQuota(c, id, quote.creditedQuota) { + return + } - payMoney := getWaffoPancakePayMoney(req.Amount, group) + payMoney := quote.payMoney if payMoney < 0.01 { c.JSON(http.StatusOK, gin.H{"message": "error", "data": "充值金额过低"}) return @@ -383,7 +397,7 @@ func RequestWaffoPancakePay(c *gin.Context) { tradeNo := fmt.Sprintf("WAFFO_PANCAKE-%d-%d-%s", id, time.Now().UnixMilli(), randstr.String(6)) topUp := &model.TopUp{ UserId: id, - Amount: normalizeWaffoPancakeTopUpAmount(req.Amount), + Amount: quote.amount, Money: payMoney, TradeNo: tradeNo, PaymentMethod: model.PaymentMethodWaffoPancake, @@ -527,6 +541,11 @@ func WaffoPancakeWebhook(c *gin.Context) { if err := model.RechargeWaffoPancake(tradeNo, model.PaymentValidationFromMajorString(event.Data.Amount, event.Data.Currency, "USD", false)); err != nil { + if errors.Is(err, model.ErrTopUpNeedsReconciliation) { + logger.LogWarn(c.Request.Context(), fmt.Sprintf("Waffo Pancake 已付款充值订单等待人工对账 trade_no=%s event_id=%s order_id=%s client_ip=%s", tradeNo, event.ID, event.Data.OrderID, c.ClientIP())) + c.String(http.StatusOK, "OK") + return + } logger.LogError(c.Request.Context(), fmt.Sprintf("Waffo Pancake 充值处理失败 trade_no=%s event_id=%s order_id=%s client_ip=%s error=%q", tradeNo, event.ID, event.Data.OrderID, c.ClientIP(), err.Error())) c.String(http.StatusInternalServerError, "retry") return diff --git a/controller/topup_waffo_pancake_test.go b/controller/topup_waffo_pancake_test.go index 097f043e..eac63848 100644 --- a/controller/topup_waffo_pancake_test.go +++ b/controller/topup_waffo_pancake_test.go @@ -14,6 +14,7 @@ import ( "github.com/MAX-API-Next/MAX-API/setting" "github.com/MAX-API-Next/MAX-API/setting/operation_setting" "github.com/gin-gonic/gin" + "github.com/shopspring/decimal" "github.com/stretchr/testify/require" "gorm.io/gorm" ) @@ -84,7 +85,7 @@ func TestFormatWaffoPancakeAmount_UsesDisplayPriceString(t *testing.T) { } } -func TestGetWaffoPancakePayMoney(t *testing.T) { +func TestBuildWaffoPancakeTopUpQuote(t *testing.T) { originalUnitPrice := setting.WaffoPancakeUnitPrice originalQuotaDisplayType := operation_setting.GetGeneralSetting().QuotaDisplayType originalDiscounts := make(map[int]float64, len(operation_setting.GetPaymentSetting().AmountDiscount)) @@ -113,40 +114,96 @@ func TestGetWaffoPancakePayMoney(t *testing.T) { amount int64 group string quotaDisplayType string - expected float64 + expectedAmount int64 + expectedQuota string + expectedMoney float64 }{ { name: "currency display applies unit price group ratio and discount", amount: 10, group: "vip", quotaDisplayType: operation_setting.QuotaDisplayTypeUSD, - expected: 24, + expectedAmount: 10, + expectedQuota: decimal.NewFromFloat(common.QuotaPerUnit).Mul(decimal.NewFromInt(10)).String(), + expectedMoney: 24, }, { name: "tokens display converts quota to display units before pricing", amount: int64(common.QuotaPerUnit * 3), group: "vip", quotaDisplayType: operation_setting.QuotaDisplayTypeTokens, - expected: 4.5, + expectedAmount: 3, + expectedQuota: decimal.NewFromFloat(common.QuotaPerUnit).Mul(decimal.NewFromInt(3)).String(), + expectedMoney: 4.5, }, { name: "non-positive discount falls back to no discount", amount: 20, group: "default", quotaDisplayType: operation_setting.QuotaDisplayTypeUSD, - expected: 50, + expectedAmount: 20, + expectedQuota: decimal.NewFromFloat(common.QuotaPerUnit).Mul(decimal.NewFromInt(20)).String(), + expectedMoney: 50, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { operation_setting.GetGeneralSetting().QuotaDisplayType = tc.quotaDisplayType - actual := getWaffoPancakePayMoney(tc.amount, tc.group) - require.InDelta(t, tc.expected, actual, 0.000001) + quote := buildWaffoPancakeTopUpQuote(tc.amount, tc.group) + require.Equal(t, tc.expectedAmount, quote.amount) + require.Equal(t, tc.expectedQuota, quote.creditedQuota.String()) + require.InDelta(t, tc.expectedMoney, quote.payMoney, 0.000001) }) } } +func TestRequestWaffoPancakeAmountPricesSubUnitTokenFromCreditedQuota(t *testing.T) { + db := setupWaffoPancakeControllerTestDB(t) + originalUnitPrice := setting.WaffoPancakeUnitPrice + originalMinTopUp := setting.WaffoPancakeMinTopUp + originalDisplayType := operation_setting.GetGeneralSetting().QuotaDisplayType + originalQuotaPerUnit := common.QuotaPerUnit + originalDiscounts := operation_setting.GetPaymentSetting().AmountDiscount + originalTopupGroupRatio := common.TopupGroupRatio2JSONString() + t.Cleanup(func() { + setting.WaffoPancakeUnitPrice = originalUnitPrice + setting.WaffoPancakeMinTopUp = originalMinTopUp + operation_setting.GetGeneralSetting().QuotaDisplayType = originalDisplayType + common.QuotaPerUnit = originalQuotaPerUnit + operation_setting.GetPaymentSetting().AmountDiscount = originalDiscounts + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(originalTopupGroupRatio)) + }) + + setting.WaffoPancakeUnitPrice = 1 + setting.WaffoPancakeMinTopUp = 1 + operation_setting.GetGeneralSetting().QuotaDisplayType = operation_setting.QuotaDisplayTypeTokens + common.QuotaPerUnit = 500000 + operation_setting.GetPaymentSetting().AmountDiscount = map[int]float64{} + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(`{"default":1}`)) + + user := &model.User{Id: 1202, Username: "pancake-token-user", Group: "default", Status: common.UserStatusEnabled} + require.NoError(t, db.Create(user).Error) + payload, err := common.Marshal(WaffoPancakePayRequest{Amount: 1}) + require.NoError(t, err) + recorder := httptest.NewRecorder() + ctx, _ := gin.CreateTestContext(recorder) + ctx.Request = httptest.NewRequest(http.MethodPost, "/api/waffo-pancake/amount", bytes.NewReader(payload)) + ctx.Request.Header.Set("Content-Type", "application/json") + ctx.Set("id", user.Id) + + RequestWaffoPancakeAmount(ctx) + + require.Equal(t, http.StatusOK, recorder.Code) + var response struct { + Message string `json:"message"` + Data string `json:"data"` + } + require.NoError(t, common.Unmarshal(recorder.Body.Bytes(), &response)) + require.Equal(t, "success", response.Message) + require.Equal(t, "1.00", response.Data) +} + func TestRequestWaffoPancakePayLeavesOrderPendingWhenSessionCreateFails(t *testing.T) { db := setupWaffoPancakeControllerTestDB(t) confirmPaymentComplianceForTest(t) @@ -157,6 +214,9 @@ func TestRequestWaffoPancakePayLeavesOrderPendingWhenSessionCreateFails(t *testi originalUnitPrice := setting.WaffoPancakeUnitPrice originalMinTopUp := setting.WaffoPancakeMinTopUp originalQuotaDisplayType := operation_setting.GetGeneralSetting().QuotaDisplayType + originalQuotaPerUnit := common.QuotaPerUnit + originalDiscounts := operation_setting.GetPaymentSetting().AmountDiscount + originalTopupGroupRatio := common.TopupGroupRatio2JSONString() t.Cleanup(func() { setting.WaffoPancakeMerchantID = originalMerchantID setting.WaffoPancakePrivateKey = originalPrivateKey @@ -164,6 +224,9 @@ func TestRequestWaffoPancakePayLeavesOrderPendingWhenSessionCreateFails(t *testi setting.WaffoPancakeUnitPrice = originalUnitPrice setting.WaffoPancakeMinTopUp = originalMinTopUp operation_setting.GetGeneralSetting().QuotaDisplayType = originalQuotaDisplayType + common.QuotaPerUnit = originalQuotaPerUnit + operation_setting.GetPaymentSetting().AmountDiscount = originalDiscounts + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(originalTopupGroupRatio)) }) setting.WaffoPancakeMerchantID = "MER_1234567890123456789012" @@ -171,7 +234,10 @@ func TestRequestWaffoPancakePayLeavesOrderPendingWhenSessionCreateFails(t *testi setting.WaffoPancakeProductID = "PROD_1234567890123456789012" setting.WaffoPancakeUnitPrice = 1 setting.WaffoPancakeMinTopUp = 1 - operation_setting.GetGeneralSetting().QuotaDisplayType = operation_setting.QuotaDisplayTypeUSD + operation_setting.GetGeneralSetting().QuotaDisplayType = operation_setting.QuotaDisplayTypeTokens + common.QuotaPerUnit = 500000 + operation_setting.GetPaymentSetting().AmountDiscount = map[int]float64{} + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(`{"default":1}`)) user := &model.User{ Id: 1201, @@ -181,7 +247,7 @@ func TestRequestWaffoPancakePayLeavesOrderPendingWhenSessionCreateFails(t *testi } require.NoError(t, db.Create(user).Error) - payload, err := common.Marshal(WaffoPancakePayRequest{Amount: 10}) + payload, err := common.Marshal(WaffoPancakePayRequest{Amount: 1}) require.NoError(t, err) recorder := httptest.NewRecorder() ctx, _ := gin.CreateTestContext(recorder) @@ -197,5 +263,7 @@ func TestRequestWaffoPancakePayLeavesOrderPendingWhenSessionCreateFails(t *testi var topUps []model.TopUp require.NoError(t, db.Find(&topUps).Error) require.Len(t, topUps, 1) + require.Equal(t, int64(1), topUps[0].Amount) + require.InDelta(t, 1, topUps[0].Money, 0.000001) require.Equal(t, common.TopUpStatusPending, topUps[0].Status) } diff --git a/controller/topup_waffo_test.go b/controller/topup_waffo_test.go new file mode 100644 index 00000000..6b63ce65 --- /dev/null +++ b/controller/topup_waffo_test.go @@ -0,0 +1,166 @@ +package controller + +import ( + "bytes" + "net/http" + "net/http/httptest" + "testing" + + "github.com/MAX-API-Next/MAX-API/common" + "github.com/MAX-API-Next/MAX-API/model" + "github.com/MAX-API-Next/MAX-API/setting" + "github.com/MAX-API-Next/MAX-API/setting/operation_setting" + "github.com/gin-gonic/gin" + "github.com/shopspring/decimal" + "github.com/stretchr/testify/require" +) + +func TestBuildWaffoTopUpQuoteKeepsPriceAndCreditConsistent(t *testing.T) { + originalDisplayType := operation_setting.GetGeneralSetting().QuotaDisplayType + originalQuotaPerUnit := common.QuotaPerUnit + originalUnitPrice := setting.WaffoUnitPrice + originalDiscounts := operation_setting.GetPaymentSetting().AmountDiscount + originalTopupGroupRatio := common.TopupGroupRatio2JSONString() + t.Cleanup(func() { + operation_setting.GetGeneralSetting().QuotaDisplayType = originalDisplayType + common.QuotaPerUnit = originalQuotaPerUnit + setting.WaffoUnitPrice = originalUnitPrice + operation_setting.GetPaymentSetting().AmountDiscount = originalDiscounts + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(originalTopupGroupRatio)) + }) + + common.QuotaPerUnit = 500000 + setting.WaffoUnitPrice = 1 + operation_setting.GetPaymentSetting().AmountDiscount = map[int]float64{} + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(`{"default":1}`)) + + tests := []struct { + name string + displayType string + amount int64 + wantStored int64 + wantQuota string + wantMoney float64 + }{ + {name: "currency", displayType: operation_setting.QuotaDisplayTypeUSD, amount: 3, wantStored: 3, wantQuota: "1500000", wantMoney: 3}, + {name: "token sub unit", displayType: operation_setting.QuotaDisplayTypeTokens, amount: 1, wantStored: 1, wantQuota: "500000", wantMoney: 1}, + {name: "token whole units", displayType: operation_setting.QuotaDisplayTypeTokens, amount: 1500000, wantStored: 3, wantQuota: "1500000", wantMoney: 3}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + operation_setting.GetGeneralSetting().QuotaDisplayType = tc.displayType + quote := buildWaffoTopUpQuote(tc.amount, "default") + require.Equal(t, tc.wantStored, quote.amount) + require.True(t, decimal.RequireFromString(tc.wantQuota).Equal(quote.creditedQuota)) + require.InDelta(t, tc.wantMoney, quote.payMoney, 0.000001) + }) + } +} + +func TestRequestWaffoAmountPricesSubUnitTokenFromCreditedQuota(t *testing.T) { + db := setupWaffoPancakeControllerTestDB(t) + originalUnitPrice := setting.WaffoUnitPrice + originalMinTopUp := setting.WaffoMinTopUp + originalDisplayType := operation_setting.GetGeneralSetting().QuotaDisplayType + originalQuotaPerUnit := common.QuotaPerUnit + originalDiscounts := operation_setting.GetPaymentSetting().AmountDiscount + originalTopupGroupRatio := common.TopupGroupRatio2JSONString() + t.Cleanup(func() { + setting.WaffoUnitPrice = originalUnitPrice + setting.WaffoMinTopUp = originalMinTopUp + operation_setting.GetGeneralSetting().QuotaDisplayType = originalDisplayType + common.QuotaPerUnit = originalQuotaPerUnit + operation_setting.GetPaymentSetting().AmountDiscount = originalDiscounts + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(originalTopupGroupRatio)) + }) + + setting.WaffoUnitPrice = 1 + setting.WaffoMinTopUp = 1 + operation_setting.GetGeneralSetting().QuotaDisplayType = operation_setting.QuotaDisplayTypeTokens + common.QuotaPerUnit = 500000 + operation_setting.GetPaymentSetting().AmountDiscount = map[int]float64{} + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(`{"default":1}`)) + + user := &model.User{Id: 1301, Username: "waffo-token-user", Group: "default", Status: common.UserStatusEnabled} + require.NoError(t, db.Create(user).Error) + payload, err := common.Marshal(WaffoPayRequest{Amount: 1}) + require.NoError(t, err) + recorder := httptest.NewRecorder() + ctx, _ := gin.CreateTestContext(recorder) + ctx.Request = httptest.NewRequest(http.MethodPost, "/api/waffo/amount", bytes.NewReader(payload)) + ctx.Request.Header.Set("Content-Type", "application/json") + ctx.Set("id", user.Id) + + RequestWaffoAmount(ctx) + + require.Equal(t, http.StatusOK, recorder.Code) + var response struct { + Message string `json:"message"` + Data string `json:"data"` + } + require.NoError(t, common.Unmarshal(recorder.Body.Bytes(), &response)) + require.Equal(t, "success", response.Message) + require.Equal(t, "1.00", response.Data) +} + +func TestRequestWaffoPayPersistsSubUnitTokenQuoteConsistently(t *testing.T) { + db := setupWaffoPancakeControllerTestDB(t) + originalEnabled := setting.WaffoEnabled + originalSandbox := setting.WaffoSandbox + originalAPIKey := setting.WaffoApiKey + originalPrivateKey := setting.WaffoPrivateKey + originalPublicCert := setting.WaffoPublicCert + originalUnitPrice := setting.WaffoUnitPrice + originalMinTopUp := setting.WaffoMinTopUp + originalDisplayType := operation_setting.GetGeneralSetting().QuotaDisplayType + originalQuotaPerUnit := common.QuotaPerUnit + originalDiscounts := operation_setting.GetPaymentSetting().AmountDiscount + originalTopupGroupRatio := common.TopupGroupRatio2JSONString() + t.Cleanup(func() { + setting.WaffoEnabled = originalEnabled + setting.WaffoSandbox = originalSandbox + setting.WaffoApiKey = originalAPIKey + setting.WaffoPrivateKey = originalPrivateKey + setting.WaffoPublicCert = originalPublicCert + setting.WaffoUnitPrice = originalUnitPrice + setting.WaffoMinTopUp = originalMinTopUp + operation_setting.GetGeneralSetting().QuotaDisplayType = originalDisplayType + common.QuotaPerUnit = originalQuotaPerUnit + operation_setting.GetPaymentSetting().AmountDiscount = originalDiscounts + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(originalTopupGroupRatio)) + }) + + setting.WaffoEnabled = true + setting.WaffoSandbox = false + setting.WaffoApiKey = "api-key" + setting.WaffoPrivateKey = "invalid-private-key" + setting.WaffoPublicCert = "invalid-public-key" + setting.WaffoUnitPrice = 1 + setting.WaffoMinTopUp = 1 + operation_setting.GetGeneralSetting().QuotaDisplayType = operation_setting.QuotaDisplayTypeTokens + common.QuotaPerUnit = 500000 + operation_setting.GetPaymentSetting().AmountDiscount = map[int]float64{} + require.NoError(t, common.UpdateTopupGroupRatioByJSONString(`{"default":1}`)) + + user := &model.User{Id: 1302, Username: "waffo-order-token-user", Group: "default", Status: common.UserStatusEnabled} + require.NoError(t, db.Create(user).Error) + payload, err := common.Marshal(WaffoPayRequest{Amount: 1}) + require.NoError(t, err) + recorder := httptest.NewRecorder() + ctx, _ := gin.CreateTestContext(recorder) + ctx.Request = httptest.NewRequest(http.MethodPost, "/api/waffo/pay", bytes.NewReader(payload)) + ctx.Request.Header.Set("Content-Type", "application/json") + ctx.Set("id", user.Id) + + RequestWaffoPay(ctx) + + require.Equal(t, http.StatusOK, recorder.Code) + require.Contains(t, recorder.Body.String(), `"data":"支付配置错误"`) + var topUps []model.TopUp + require.NoError(t, db.Find(&topUps).Error) + require.Len(t, topUps, 1) + require.Equal(t, int64(1), topUps[0].Amount) + require.InDelta(t, 1, topUps[0].Money, 0.000001) + require.Equal(t, common.TopUpStatusFailed, topUps[0].Status) +} diff --git a/docker-compose.yml b/docker-compose.yml index e0c5a420..05abd152 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -33,7 +33,7 @@ services: - ERROR_LOG_ENABLED=true # 是否启用错误日志记录 (Whether to enable error log recording) - BATCH_UPDATE_ENABLED=true # 是否启用批量更新 (Whether to enable batch update) - NODE_NAME=max-api-node-1 # 节点名称,用于审计日志中标识节点身份;多节点/容器部署时建议设置 (Node name used in audit logs; recommended when running multiple instances or in containers) -# - STREAMING_TIMEOUT=300 # 流模式无响应超时时间,单位秒,默认120秒,如果出现空补全可以尝试改为更大值 (Streaming timeout in seconds, default is 120s. Increase if experiencing empty completions) +# - STREAMING_TIMEOUT=300 # 流模式无响应超时时间,单位秒,默认300秒,如果出现空补全可以尝试改为更大值 (Streaming timeout in seconds, default is 300s. Increase if experiencing empty completions) # - SESSION_SECRET=random_string # 多机部署时设置,必须修改这个随机字符串!! (multi-node deployment, set this to a random string!!!!!!!) # - SYNC_FREQUENCY=60 # Uncomment if regular database syncing is needed # - GOOGLE_ANALYTICS_ID=G-XXXXXXXXXX # Google Analytics 的测量 ID (Google Analytics Measurement ID) diff --git a/dto/notify.go b/dto/notify.go index b75cec70..a3ac24e5 100644 --- a/dto/notify.go +++ b/dto/notify.go @@ -13,6 +13,7 @@ const ( NotifyTypeQuotaExceed = "quota_exceed" NotifyTypeChannelUpdate = "channel_update" NotifyTypeChannelTest = "channel_test" + NotifyTypeSmartOpsAlert = "smart_ops_alert" ) func NewNotify(t string, title string, content string, values []interface{}) Notify { diff --git a/dto/openai_request.go b/dto/openai_request.go index 8220ba7d..9a4701cb 100644 --- a/dto/openai_request.go +++ b/dto/openai_request.go @@ -866,15 +866,19 @@ type OpenAIResponsesRequest struct { Include json.RawMessage `json:"include,omitempty"` // 在后台运行推理,暂时还不支持依赖的接口 // Background json.RawMessage `json:"background,omitempty"` - Conversation json.RawMessage `json:"conversation,omitempty"` - ContextManagement json.RawMessage `json:"context_management,omitempty"` - Instructions json.RawMessage `json:"instructions,omitempty"` - MaxOutputTokens *uint `json:"max_output_tokens,omitempty"` - TopLogProbs *int `json:"top_logprobs,omitempty"` - Metadata json.RawMessage `json:"metadata,omitempty"` - ParallelToolCalls json.RawMessage `json:"parallel_tool_calls,omitempty"` - PreviousResponseID string `json:"previous_response_id,omitempty"` - Reasoning *Reasoning `json:"reasoning,omitempty"` + Conversation json.RawMessage `json:"conversation,omitempty"` + ContextManagement json.RawMessage `json:"context_management,omitempty"` + Instructions json.RawMessage `json:"instructions,omitempty"` + MaxOutputTokens *uint `json:"max_output_tokens,omitempty"` + TopLogProbs *int `json:"top_logprobs,omitempty"` + Metadata json.RawMessage `json:"metadata,omitempty"` + ParallelToolCalls json.RawMessage `json:"parallel_tool_calls,omitempty"` + // Accepted by some OpenAI-compatible Responses providers even though the + // official OpenAI Responses API does not currently document these fields. + FrequencyPenalty *float64 `json:"frequency_penalty,omitempty"` + PresencePenalty *float64 `json:"presence_penalty,omitempty"` + PreviousResponseID string `json:"previous_response_id,omitempty"` + Reasoning *Reasoning `json:"reasoning,omitempty"` // ServiceTier specifies upstream service level and may affect billing. // This field is filtered by default and can be enabled via channel setting allow_service_tier. ServiceTier string `json:"service_tier,omitempty"` diff --git a/dto/openai_request_zero_value_test.go b/dto/openai_request_zero_value_test.go index 2b7a4739..c37c4076 100644 --- a/dto/openai_request_zero_value_test.go +++ b/dto/openai_request_zero_value_test.go @@ -150,7 +150,9 @@ func TestOpenAIResponsesRequestPreserveExplicitZeroValues(t *testing.T) { "max_output_tokens":0, "max_tool_calls":0, "stream":false, - "top_p":0 + "top_p":0, + "frequency_penalty":0, + "presence_penalty":0 }`) var req OpenAIResponsesRequest @@ -164,6 +166,18 @@ func TestOpenAIResponsesRequestPreserveExplicitZeroValues(t *testing.T) { require.True(t, gjson.GetBytes(encoded, "max_tool_calls").Exists()) require.True(t, gjson.GetBytes(encoded, "stream").Exists()) require.True(t, gjson.GetBytes(encoded, "top_p").Exists()) + require.True(t, gjson.GetBytes(encoded, "frequency_penalty").Exists()) + require.True(t, gjson.GetBytes(encoded, "presence_penalty").Exists()) +} + +func TestOpenAIResponsesRequestRejectsNonNumericPenalties(t *testing.T) { + for _, raw := range []string{ + `{"model":"gpt-4.1","frequency_penalty":{}}`, + `{"model":"gpt-4.1","presence_penalty":"high"}`, + } { + var req OpenAIResponsesRequest + require.Error(t, common.Unmarshal([]byte(raw), &req)) + } } func TestOpenAIResponsesRequestPreserveQwenThinkingBudget(t *testing.T) { diff --git a/go.mod b/go.mod index 04ad5da9..5523f932 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,6 @@ module github.com/MAX-API-Next/MAX-API -// +heroku goVersion go1.18 +// +heroku goVersion go1.25 go 1.25.1 require ( diff --git a/main.go b/main.go index e8ee0e2a..6dfae480 100644 --- a/main.go +++ b/main.go @@ -128,6 +128,9 @@ func main() { // System instance heartbeat for multi-node operations visibility service.StartSystemInstanceReporter() + // Proactively detect sustained system resource pressure and notify admins. + service.StartSmartOpsAlertMonitor() + // Persistent system maintenance task runner service.StartSystemTaskRunner() authFlowCleanupDone := service.StartAuthFlowCleanupWithContext(backgroundCtx) @@ -251,6 +254,9 @@ func main() { if err := common.StopSystemMonitor(ctx); err != nil { common.SysError(fmt.Sprintf("timed out stopping system monitor: %v", err)) } + if err := service.StopSmartOpsAlertMonitor(ctx); err != nil { + common.SysError(fmt.Sprintf("timed out stopping smart ops alert monitor: %v", err)) + } if err := middleware.StopInMemoryRateLimiter(ctx); err != nil { common.SysError(fmt.Sprintf("timed out stopping in-memory rate limiter: %v", err)) } diff --git a/middleware/auth.go b/middleware/auth.go index 864dca40..f7282904 100644 --- a/middleware/auth.go +++ b/middleware/auth.go @@ -391,7 +391,8 @@ func TokenAuth() func(c *gin.Context) { } } // gemini api 从query中获取key - if strings.HasPrefix(c.Request.URL.Path, "/v1beta/models") || + if c.Request.URL.Path == "/v1/models" || + strings.HasPrefix(c.Request.URL.Path, "/v1beta/models") || strings.HasPrefix(c.Request.URL.Path, "/v1beta/openai/models") || strings.HasPrefix(c.Request.URL.Path, "/v1/models/") { skKey := c.Query("key") diff --git a/middleware/distributor.go b/middleware/distributor.go index 0b756482..d99f15bb 100644 --- a/middleware/distributor.go +++ b/middleware/distributor.go @@ -123,7 +123,7 @@ func Distribute() func(c *gin.Context) { channelSupportsRequestPath(preferred, c.Request.URL.Path) { if group, groupIndex, matched := findAffinityRouteGroup(routePlan, modelRequest.Model, preferred.Id, model.IsChannelEnabledForGroupModel); matched { selectGroup = group - common.SetContextKey(c, constant.ContextKeyAutoGroup, group) + service.SetSelectedRoutingGroup(c, group) channel = preferred affinityUsable = true service.MarkChannelAffinityUsed(c, group, preferred.Id) @@ -140,7 +140,7 @@ func Distribute() func(c *gin.Context) { for _, g := range autoGroups { if model.IsChannelEnabledForGroupModel(g, modelRequest.Model, preferred.Id) { selectGroup = g - common.SetContextKey(c, constant.ContextKeyAutoGroup, g) + service.SetSelectedRoutingGroup(c, g) channel = preferred affinityUsable = true service.MarkChannelAffinityUsed(c, g, preferred.Id) diff --git a/model/channel_performance.go b/model/channel_performance.go new file mode 100644 index 00000000..cd99cc8d --- /dev/null +++ b/model/channel_performance.go @@ -0,0 +1,238 @@ +package model + +import ( + "context" + "errors" + "fmt" + + "gorm.io/gorm" +) + +const ( + LegacyRetryMetricStateAvailable = "available" + LegacyRetryMetricStatePending = "pending" + LegacyRetryMetricStateUnavailable = "unavailable" +) + +const ( + legacyViolationFeePattern = `%"violation_fee":true%` + legacyTaskAdjustmentPattern = `%"pre_consumed_quota":%` + legacyTaskSubmissionPattern = `%"is_task":true%` + legacyViolationFeeContent = "Violation fee charged" +) + +// ChannelPerformanceQuery describes the bounded, read-only legacy log query +// used by Smart Operations Center. It deliberately does not model a Relay +// Attempt; a future Attempt reader can implement the service seam separately. +type ChannelPerformanceQuery struct { + StartAt int64 + EndAt int64 + Hours int + Limit int + ChannelID int + ModelName string + Group string +} + +// LegacyChannelPerformanceRow is an aggregate projection from LOG_DB. Keep +// this projection narrow so channel credentials, prompts and responses never +// enter the operations surface. +type LegacyChannelPerformanceRow struct { + ChannelID int `gorm:"column:channel_id"` + ModelName string `gorm:"column:model_name"` + EffectiveGroup string `gorm:"column:effective_group"` + ObservedCount int64 `gorm:"column:observed_count"` + ConsumeLogCount int64 `gorm:"column:consume_log_count"` + ErrorLogCount int64 `gorm:"column:error_log_count"` + ConsumedQuota int64 `gorm:"column:consumed_quota"` + RetryLogCount int64 `gorm:"column:retry_log_count"` + LatencySampleCount int64 `gorm:"column:latency_sample_count"` + TotalUseTimeSeconds int64 `gorm:"column:total_use_time_seconds"` + LastObservedAt int64 `gorm:"column:last_observed_at"` +} + +type LegacyChannelPerformanceSummary struct { + ChannelCount int64 `gorm:"column:channel_count"` + ObservedCount int64 `gorm:"column:observed_count"` + ConsumeLogCount int64 `gorm:"column:consume_log_count"` + ErrorLogCount int64 `gorm:"column:error_log_count"` + ConsumedQuota int64 `gorm:"column:consumed_quota"` + RetryLogCount int64 `gorm:"column:retry_log_count"` + LatencySampleCount int64 `gorm:"column:latency_sample_count"` + TotalUseTimeSeconds int64 `gorm:"column:total_use_time_seconds"` + LastObservedAt int64 `gorm:"column:last_observed_at"` +} + +type LegacyChannelPerformanceResult struct { + Rows []LegacyChannelPerformanceRow + Summary LegacyChannelPerformanceSummary + RetryMetricsAvailable bool + RetryMetricState string +} + +// ChannelPerformanceChannel is the safe metadata projection joined in the +// application after LOG_DB aggregation. It intentionally excludes Key, +// BaseURL, overrides, settings and all other secret-bearing fields. +type ChannelPerformanceChannel struct { + ID int `gorm:"column:id"` + Name string `gorm:"column:name"` + Type int `gorm:"column:type"` + Status int `gorm:"column:status"` + ResponseTime int `gorm:"column:response_time"` + TestTime int64 `gorm:"column:test_time"` +} + +// QueryLegacyChannelPerformance reads only existing log rows. The bounded row +// query returns one extra row for truncation detection, while the independent +// summary query always covers every matching eligible log. +func QueryLegacyChannelPerformance(ctx context.Context, query ChannelPerformanceQuery) (LegacyChannelPerformanceResult, error) { + if LOG_DB == nil { + return LegacyChannelPerformanceResult{}, fmt.Errorf("log database is not initialized") + } + + limit := query.Limit + if limit <= 0 { + limit = 100 + } + if limit > 200 { + limit = 200 + } + + // group is reserved by all supported databases. logGroupCol is selected by + // initCol for the configured LOG_DB dialect. + groupColumn := logGroupCol + if groupColumn == "" { + groupColumn = commonGroupCol + } + + retryMetricState := legacyChannelPerformanceRetryMetricState() + retryMetricsAvailable := retryMetricState == LegacyRetryMetricStateAvailable + rowSelect, rowArgs := legacyChannelPerformanceSelect(groupColumn, true, retryMetricsAvailable) + + var rows []LegacyChannelPerformanceRow + err := legacyChannelPerformanceQuery(ctx, query, groupColumn). + Select(rowSelect, rowArgs...). + Group("channel_id, model_name, " + groupColumn). + Order("observed_count DESC, last_observed_at DESC"). + Limit(limit + 1). + Scan(&rows).Error + if err != nil { + return LegacyChannelPerformanceResult{}, err + } + + summarySelect, summaryArgs := legacyChannelPerformanceSelect(groupColumn, false, retryMetricsAvailable) + var summary LegacyChannelPerformanceSummary + if err := legacyChannelPerformanceQuery(ctx, query, groupColumn). + Select(summarySelect, summaryArgs...). + Scan(&summary).Error; err != nil { + return LegacyChannelPerformanceResult{}, err + } + + return LegacyChannelPerformanceResult{ + Rows: rows, + Summary: summary, + RetryMetricsAvailable: retryMetricsAvailable, + RetryMetricState: retryMetricState, + }, nil +} + +func legacyChannelPerformanceRetryMetricState() string { + if err := ensureLogRetryMarkerBackfillCompletedForRead(); err == nil { + return LegacyRetryMetricStateAvailable + } else if errors.Is(err, ErrLogRetryMarkerBackfillIncomplete) { + return LegacyRetryMetricStatePending + } + return LegacyRetryMetricStateUnavailable +} + +func legacyChannelPerformanceQuery(ctx context.Context, query ChannelPerformanceQuery, groupColumn string) *gorm.DB { + tx := LOG_DB.WithContext(ctx).Table("logs") + tx = tx.Where("created_at >= ? AND created_at <= ?", query.StartAt, query.EndAt) + tx = tx.Where("type IN ?", []int{LogTypeConsume, LogTypeError}) + tx = tx.Where("channel_id > ?", 0).Where("model_name <> ?", "") + // Channel tests are probes, not production traffic. Billing-only consume + // records are accounting evidence, not successful Relay attempts. Keep the + // accounting rows intact and exclude them only from this read projection. + tx = tx.Where("(content IS NULL OR content <> ?)", "模型测试"). + Where("(token_name IS NULL OR token_name <> ?)", "模型测试"). + Where( + "(type <> ? OR ((other IS NULL OR (other NOT LIKE ? AND other NOT LIKE ?)) AND (content IS NULL OR content <> ?)))", + LogTypeConsume, + legacyViolationFeePattern, + legacyTaskAdjustmentPattern, + legacyViolationFeeContent, + ) + if query.ChannelID > 0 { + tx = tx.Where("channel_id = ?", query.ChannelID) + } + if query.ModelName != "" { + tx = tx.Where("model_name = ?", query.ModelName) + } + if query.Group != "" { + tx = tx.Where(groupColumn+" = ?", query.Group) + } + return tx +} + +func legacyChannelPerformanceSelect(groupColumn string, includeDimensions bool, retryMetricsAvailable bool) (string, []any) { + prefix := "COUNT(DISTINCT channel_id) AS channel_count, " + if includeDimensions { + prefix = fmt.Sprintf("channel_id, model_name, %s AS effective_group, ", groupColumn) + } + + retryExpr := "0 AS retry_log_count, " + args := []any{LogTypeConsume, LogTypeError, LogTypeConsume} + if retryMetricsAvailable { + retryExpr = "SUM(CASE WHEN is_retry = ? OR is_error_retry = ? OR is_empty_retry = ? THEN 1 ELSE 0 END) AS retry_log_count, " + args = append(args, true, true, true) + } + // consumed_quota is an operational projection over the same eligible + // Consume logs as the performance row. It is not a complete billing ledger + // total because known accounting-only adjustments are excluded upstream. + selectClause := prefix + + "COUNT(*) AS observed_count, " + + "SUM(CASE WHEN type = ? THEN 1 ELSE 0 END) AS consume_log_count, " + + "SUM(CASE WHEN type = ? THEN 1 ELSE 0 END) AS error_log_count, " + + "SUM(CASE WHEN type = ? THEN quota ELSE 0 END) AS consumed_quota, " + + retryExpr + if !retryMetricsAvailable { + return selectClause + + "0 AS total_use_time_seconds, " + + "0 AS latency_sample_count, " + + "MAX(created_at) AS last_observed_at", args + } + + args = append(args, + LogTypeConsume, legacyTaskSubmissionPattern, false, false, false, + LogTypeConsume, legacyTaskSubmissionPattern, false, false, false, + ) + return selectClause + + "SUM(CASE WHEN (type <> ? OR other IS NULL OR other NOT LIKE ?) AND is_retry = ? AND is_error_retry = ? AND is_empty_retry = ? THEN use_time ELSE 0 END) AS total_use_time_seconds, " + + "SUM(CASE WHEN (type <> ? OR other IS NULL OR other NOT LIKE ?) AND is_retry = ? AND is_error_retry = ? AND is_empty_retry = ? THEN 1 ELSE 0 END) AS latency_sample_count, " + + "MAX(created_at) AS last_observed_at", args +} + +// GetChannelPerformanceChannels returns only the metadata required to label +// aggregate rows. It is intentionally a separate query because LOG_DB may be +// configured as a different database from the primary application database. +func GetChannelPerformanceChannels(ctx context.Context, ids []int) (map[int]ChannelPerformanceChannel, error) { + result := make(map[int]ChannelPerformanceChannel, len(ids)) + if len(ids) == 0 { + return result, nil + } + if DB == nil { + return nil, fmt.Errorf("primary database is not initialized") + } + + var channels []ChannelPerformanceChannel + err := DB.WithContext(ctx).Model(&Channel{}). + Select("id, name, type, status, response_time, test_time"). + Where("id IN ?", ids).Find(&channels).Error + if err != nil { + return nil, err + } + for _, channel := range channels { + result[channel.ID] = channel + } + return result, nil +} diff --git a/model/channel_performance_test.go b/model/channel_performance_test.go new file mode 100644 index 00000000..162bc95a --- /dev/null +++ b/model/channel_performance_test.go @@ -0,0 +1,207 @@ +package model + +import ( + "context" + "testing" + + "github.com/MAX-API-Next/MAX-API/common" + "github.com/glebarez/sqlite" + "github.com/stretchr/testify/require" + "gorm.io/gorm" +) + +func setupChannelPerformanceTestDB(t *testing.T, retryMetricsReady bool) *gorm.DB { + t.Helper() + + previousDB, previousLogDB := DB, LOG_DB + previousSQLite, previousPostgres, previousMySQL := common.UsingSQLite, common.UsingPostgreSQL, common.UsingMySQL + previousLogSQLType := common.LogSqlType + t.Cleanup(func() { + DB, LOG_DB = previousDB, previousLogDB + common.UsingSQLite, common.UsingPostgreSQL, common.UsingMySQL = previousSQLite, previousPostgres, previousMySQL + common.LogSqlType = previousLogSQLType + initCol() + }) + + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + DB, LOG_DB = db, db + common.UsingSQLite, common.UsingPostgreSQL, common.UsingMySQL = true, false, false + common.LogSqlType = common.DatabaseTypeSQLite + initCol() + require.NoError(t, db.AutoMigrate(&Log{}, &Channel{}, &Option{})) + if retryMetricsReady { + require.NoError(t, markLogRetryMarkerBackfillCompleted()) + } + return db +} + +func TestInitColUsesIndependentLogDatabaseDialect(t *testing.T) { + previousSQLite, previousPostgres, previousMySQL := common.UsingSQLite, common.UsingPostgreSQL, common.UsingMySQL + previousLogSQLType := common.LogSqlType + t.Cleanup(func() { + common.UsingSQLite, common.UsingPostgreSQL, common.UsingMySQL = previousSQLite, previousPostgres, previousMySQL + common.LogSqlType = previousLogSQLType + initCol() + }) + t.Setenv("LOG_SQL_DSN", "separate-log-database") + + common.UsingSQLite, common.UsingPostgreSQL, common.UsingMySQL = false, true, false + common.LogSqlType = common.DatabaseTypeMySQL + initCol() + require.Equal(t, `"group"`, commonGroupCol) + require.Equal(t, "`group`", logGroupCol) + require.Equal(t, "`key`", logKeyCol) + + common.UsingSQLite, common.UsingPostgreSQL, common.UsingMySQL = false, false, true + common.LogSqlType = common.DatabaseTypePostgreSQL + initCol() + require.Equal(t, "`group`", commonGroupCol) + require.Equal(t, `"group"`, logGroupCol) + require.Equal(t, `"key"`, logKeyCol) +} + +func TestQueryLegacyChannelPerformanceExcludesNonAttemptBillingLogs(t *testing.T) { + db := setupChannelPerformanceTestDB(t, true) + + now := int64(1_700_000_000) + require.NoError(t, db.Create(&Channel{Id: 7, Name: "Primary", Type: 1, Status: 1, ResponseTime: 240, TestTime: now}).Error) + require.NoError(t, db.Create(&[]Log{ + {CreatedAt: now - 20, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 120, UseTime: 9, Other: `{"retry_log":true}`}, + {CreatedAt: now - 10, Type: LogTypeError, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 999, UseTime: 1}, + {CreatedAt: now - 9, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 50, UseTime: 5, Content: "Violation fee charged", Other: `{"violation_fee":true}`}, + {CreatedAt: now - 8, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 40, Other: `{"task_id":"task-1","pre_consumed_quota":10,"actual_quota":15}`}, + {CreatedAt: now - 7, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 80, Other: `{"is_task":true}`}, + {CreatedAt: now - 5, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 70, TokenName: "模型测试"}, + {CreatedAt: now - 4, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 60, Content: "模型测试"}, + }).Error) + + result, err := QueryLegacyChannelPerformance(context.Background(), ChannelPerformanceQuery{ + StartAt: now - 100, + EndAt: now, + Limit: 10, + }) + require.NoError(t, err) + require.True(t, result.RetryMetricsAvailable) + require.Equal(t, LegacyRetryMetricStateAvailable, result.RetryMetricState) + require.Len(t, result.Rows, 1) + row := result.Rows[0] + require.Equal(t, 7, row.ChannelID) + require.Equal(t, "alpha", row.ModelName) + require.Equal(t, "prod", row.EffectiveGroup) + require.EqualValues(t, 3, row.ObservedCount) + require.EqualValues(t, 2, row.ConsumeLogCount) + require.EqualValues(t, 1, row.ErrorLogCount) + require.EqualValues(t, 200, row.ConsumedQuota) + require.EqualValues(t, 1, row.RetryLogCount) + require.EqualValues(t, 1, row.LatencySampleCount) + require.EqualValues(t, 1, row.TotalUseTimeSeconds) + require.EqualValues(t, 3, result.Summary.ObservedCount) + require.EqualValues(t, 2, result.Summary.ConsumeLogCount) + require.EqualValues(t, 1, result.Summary.ErrorLogCount) + require.EqualValues(t, 200, result.Summary.ConsumedQuota) + require.EqualValues(t, 1, result.Summary.LatencySampleCount) + require.EqualValues(t, 1, result.Summary.TotalUseTimeSeconds) + require.EqualValues(t, 1, result.Summary.ChannelCount) + require.EqualValues(t, now-7, result.Summary.LastObservedAt) + + metadata, err := GetChannelPerformanceChannels(context.Background(), []int{7}) + require.NoError(t, err) + require.Equal(t, "Primary", metadata[7].Name) + require.Equal(t, 240, metadata[7].ResponseTime) +} + +func TestQueryLegacyChannelPerformanceSummaryIsNotLimitedWithRows(t *testing.T) { + db := setupChannelPerformanceTestDB(t, true) + now := int64(1_700_000_000) + logs := make([]Log, 0, 3) + for channelID := 1; channelID <= 3; channelID++ { + logs = append(logs, Log{ + CreatedAt: now - int64(channelID), Type: LogTypeConsume, + ModelName: "alpha", ChannelId: channelID, Group: "prod", Quota: channelID * 10, UseTime: channelID, + }) + } + require.NoError(t, db.Create(&logs).Error) + + result, err := QueryLegacyChannelPerformance(context.Background(), ChannelPerformanceQuery{ + StartAt: now - 100, + EndAt: now, + Limit: 1, + }) + require.NoError(t, err) + require.Len(t, result.Rows, 2, "the row query should retain one extra row for truncation detection") + require.EqualValues(t, 3, result.Summary.ChannelCount) + require.EqualValues(t, 3, result.Summary.ObservedCount) + require.EqualValues(t, 3, result.Summary.ConsumeLogCount) + require.EqualValues(t, 60, result.Summary.ConsumedQuota) + require.EqualValues(t, 3, result.Summary.LatencySampleCount) + require.EqualValues(t, 6, result.Summary.TotalUseTimeSeconds) + require.EqualValues(t, now-1, result.Summary.LastObservedAt) +} + +func TestQueryLegacyChannelPerformancePreservesSignedConsumeQuota(t *testing.T) { + db := setupChannelPerformanceTestDB(t, true) + now := int64(1_700_000_000) + require.NoError(t, db.Create(&[]Log{ + {CreatedAt: now - 3, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 100}, + {CreatedAt: now - 2, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 0}, + {CreatedAt: now - 1, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: -25}, + {CreatedAt: now - 1, Type: LogTypeError, ModelName: "alpha", ChannelId: 7, Group: "prod", Quota: 500}, + }).Error) + + result, err := QueryLegacyChannelPerformance(context.Background(), ChannelPerformanceQuery{ + StartAt: now - 100, + EndAt: now, + Limit: 10, + }) + require.NoError(t, err) + require.Len(t, result.Rows, 1) + require.EqualValues(t, 75, result.Rows[0].ConsumedQuota) + require.EqualValues(t, 75, result.Summary.ConsumedQuota) +} + +func TestQueryLegacyChannelPerformanceMarksRetryMetricsUnavailableBeforeBackfill(t *testing.T) { + db := setupChannelPerformanceTestDB(t, false) + now := int64(1_700_000_000) + log := Log{ + CreatedAt: now - 1, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 7, + Group: "prod", UseTime: 5, Other: `{"retry_log":true}`, + } + require.NoError(t, db.Create(&log).Error) + require.NoError(t, db.Model(&Log{}).Where("id = ?", log.Id).Updates(map[string]interface{}{ + "is_retry": false, "is_error_retry": false, "is_empty_retry": false, + }).Error) + + result, err := QueryLegacyChannelPerformance(context.Background(), ChannelPerformanceQuery{ + StartAt: now - 100, + EndAt: now, + Limit: 10, + }) + require.NoError(t, err) + require.False(t, result.RetryMetricsAvailable) + require.Equal(t, LegacyRetryMetricStatePending, result.RetryMetricState) + require.Len(t, result.Rows, 1) + require.Zero(t, result.Rows[0].RetryLogCount) + require.Zero(t, result.Rows[0].LatencySampleCount) + require.Zero(t, result.Rows[0].TotalUseTimeSeconds) + require.Zero(t, result.Summary.RetryLogCount) + require.Zero(t, result.Summary.LatencySampleCount) + require.Zero(t, result.Summary.TotalUseTimeSeconds) +} + +func TestQueryLegacyChannelPerformanceReturnsZeroSummaryWithoutRows(t *testing.T) { + setupChannelPerformanceTestDB(t, true) + + result, err := QueryLegacyChannelPerformance(context.Background(), ChannelPerformanceQuery{ + StartAt: 1_700_000_000, + EndAt: 1_700_000_100, + Limit: 10, + }) + require.NoError(t, err) + require.Empty(t, result.Rows) + require.Zero(t, result.Summary.ChannelCount) + require.Zero(t, result.Summary.ObservedCount) + require.Zero(t, result.Summary.ConsumedQuota) + require.Zero(t, result.Summary.LatencySampleCount) + require.Zero(t, result.Summary.LastObservedAt) +} diff --git a/model/main.go b/model/main.go index e336bd62..84592406 100644 --- a/model/main.go +++ b/model/main.go @@ -84,9 +84,12 @@ func initCol() { case common.DatabaseTypePostgreSQL: logGroupCol = `"group"` logKeyCol = `"key"` + case common.DatabaseTypeMySQL, common.DatabaseTypeSQLite: + logGroupCol = "`group`" + logKeyCol = "`key`" default: - logGroupCol = commonGroupCol - logKeyCol = commonKeyCol + logGroupCol = "`group`" + logKeyCol = "`key`" } } else { // LOG_SQL_DSN 为空时,日志数据库与主数据库相同 @@ -345,6 +348,7 @@ func migrateDB() error { &UserOAuthBinding{}, &AuthFlow{}, &PerfMetric{}, + &PerfMetricFlushReceipt{}, &SystemTask{}, &SystemInstance{}, ) diff --git a/model/model_performance.go b/model/model_performance.go new file mode 100644 index 00000000..9a1ee067 --- /dev/null +++ b/model/model_performance.go @@ -0,0 +1,147 @@ +package model + +import ( + "context" + "fmt" +) + +// ModelPerformanceQuery describes the bounded, read-only legacy log query +// used by Smart Operations Center to aggregate production evidence by model. +type ModelPerformanceQuery struct { + StartAt int64 + EndAt int64 + Hours int + Limit int + ModelName string + Group string +} + +type LegacyModelPerformanceRow struct { + ModelName string `gorm:"column:model_name"` + ChannelCount int64 `gorm:"column:channel_count"` + ObservedCount int64 `gorm:"column:observed_count"` + ConsumeLogCount int64 `gorm:"column:consume_log_count"` + ErrorLogCount int64 `gorm:"column:error_log_count"` + ConsumedQuota int64 `gorm:"column:consumed_quota"` + RetryLogCount int64 `gorm:"column:retry_log_count"` + LatencySampleCount int64 `gorm:"column:latency_sample_count"` + TotalUseTimeSeconds int64 `gorm:"column:total_use_time_seconds"` + LastObservedAt int64 `gorm:"column:last_observed_at"` +} + +type LegacyModelPerformanceSummary struct { + ModelCount int64 `gorm:"column:model_count"` + ChannelCount int64 `gorm:"column:channel_count"` + ObservedCount int64 `gorm:"column:observed_count"` + ConsumeLogCount int64 `gorm:"column:consume_log_count"` + ErrorLogCount int64 `gorm:"column:error_log_count"` + ConsumedQuota int64 `gorm:"column:consumed_quota"` + RetryLogCount int64 `gorm:"column:retry_log_count"` + LatencySampleCount int64 `gorm:"column:latency_sample_count"` + TotalUseTimeSeconds int64 `gorm:"column:total_use_time_seconds"` + LastObservedAt int64 `gorm:"column:last_observed_at"` +} + +type LegacyModelPerformanceResult struct { + Rows []LegacyModelPerformanceRow + Summary LegacyModelPerformanceSummary + RetryMetricsAvailable bool + RetryMetricState string +} + +// QueryLegacyModelPerformance reads existing log rows and aggregates them by +// model. It intentionally shares the channel-performance eligibility rules so +// both views exclude probes, billing adjustments and known non-attempt logs in +// the same way. +func QueryLegacyModelPerformance(ctx context.Context, query ModelPerformanceQuery) (LegacyModelPerformanceResult, error) { + if LOG_DB == nil { + return LegacyModelPerformanceResult{}, fmt.Errorf("log database is not initialized") + } + + limit := query.Limit + if limit <= 0 { + limit = 100 + } + if limit > 200 { + limit = 200 + } + + groupColumn := logGroupCol + if groupColumn == "" { + groupColumn = commonGroupCol + } + + retryMetricState := legacyChannelPerformanceRetryMetricState() + retryMetricsAvailable := retryMetricState == LegacyRetryMetricStateAvailable + rowSelect, rowArgs := legacyModelPerformanceSelect(true, retryMetricsAvailable) + baseQuery := ChannelPerformanceQuery{ + StartAt: query.StartAt, + EndAt: query.EndAt, + ModelName: query.ModelName, + Group: query.Group, + } + + var rows []LegacyModelPerformanceRow + err := legacyChannelPerformanceQuery(ctx, baseQuery, groupColumn). + Select(rowSelect, rowArgs...). + Group("model_name"). + Order("observed_count DESC, last_observed_at DESC, model_name ASC"). + Limit(limit + 1). + Scan(&rows).Error + if err != nil { + return LegacyModelPerformanceResult{}, err + } + + summarySelect, summaryArgs := legacyModelPerformanceSelect(false, retryMetricsAvailable) + var summary LegacyModelPerformanceSummary + if err := legacyChannelPerformanceQuery(ctx, baseQuery, groupColumn). + Select(summarySelect, summaryArgs...). + Scan(&summary).Error; err != nil { + return LegacyModelPerformanceResult{}, err + } + + return LegacyModelPerformanceResult{ + Rows: rows, + Summary: summary, + RetryMetricsAvailable: retryMetricsAvailable, + RetryMetricState: retryMetricState, + }, nil +} + +func legacyModelPerformanceSelect(includeModel bool, retryMetricsAvailable bool) (string, []any) { + prefix := "COUNT(DISTINCT model_name) AS model_count, COUNT(DISTINCT channel_id) AS channel_count, " + if includeModel { + prefix = "model_name, COUNT(DISTINCT channel_id) AS channel_count, " + } + + retryExpr := "0 AS retry_log_count, " + args := []any{LogTypeConsume, LogTypeError, LogTypeConsume} + if retryMetricsAvailable { + retryExpr = "SUM(CASE WHEN is_retry = ? OR is_error_retry = ? OR is_empty_retry = ? THEN 1 ELSE 0 END) AS retry_log_count, " + args = append(args, true, true, true) + } + + // Keep quota on the same eligible-log projection as channel performance; + // this operational total intentionally excludes accounting-only rows. + selectClause := prefix + + "COUNT(*) AS observed_count, " + + "SUM(CASE WHEN type = ? THEN 1 ELSE 0 END) AS consume_log_count, " + + "SUM(CASE WHEN type = ? THEN 1 ELSE 0 END) AS error_log_count, " + + "SUM(CASE WHEN type = ? THEN quota ELSE 0 END) AS consumed_quota, " + + retryExpr + if !retryMetricsAvailable { + return selectClause + + "0 AS total_use_time_seconds, " + + "0 AS latency_sample_count, " + + "MAX(created_at) AS last_observed_at", args + } + + args = append(args, + LogTypeConsume, legacyTaskSubmissionPattern, false, false, false, + LogTypeConsume, legacyTaskSubmissionPattern, false, false, false, + ) + return selectClause + + "SUM(CASE WHEN (type <> ? OR other IS NULL OR other NOT LIKE ?) AND is_retry = ? AND is_error_retry = ? AND is_empty_retry = ? THEN use_time ELSE 0 END) AS total_use_time_seconds, " + + "SUM(CASE WHEN (type <> ? OR other IS NULL OR other NOT LIKE ?) AND is_retry = ? AND is_error_retry = ? AND is_empty_retry = ? THEN 1 ELSE 0 END) AS latency_sample_count, " + + "MAX(created_at) AS last_observed_at", args +} diff --git a/model/model_performance_test.go b/model/model_performance_test.go new file mode 100644 index 00000000..4f358f47 --- /dev/null +++ b/model/model_performance_test.go @@ -0,0 +1,101 @@ +package model + +import ( + "context" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestQueryLegacyModelPerformanceAggregatesAcrossChannels(t *testing.T) { + db := setupChannelPerformanceTestDB(t, true) + now := int64(1_700_000_000) + + require.NoError(t, db.Create(&[]Log{ + {CreatedAt: now - 20, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 1, Group: "prod", Quota: 120, UseTime: 2}, + {CreatedAt: now - 19, Type: LogTypeError, ModelName: "alpha", ChannelId: 1, Group: "prod", Quota: 999}, + {CreatedAt: now - 18, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 2, Group: "prod", Quota: 80, UseTime: 4, Other: `{"retry_log":true}`}, + {CreatedAt: now - 10, Type: LogTypeConsume, ModelName: "beta", ChannelId: 2, Group: "prod", Quota: 50, UseTime: 3}, + {CreatedAt: now - 9, Type: LogTypeConsume, ModelName: "ignored", ChannelId: 0, Group: "prod", Quota: 1000, UseTime: 8}, + }).Error) + + result, err := QueryLegacyModelPerformance(context.Background(), ModelPerformanceQuery{ + StartAt: now - 100, + EndAt: now, + Limit: 10, + }) + require.NoError(t, err) + require.True(t, result.RetryMetricsAvailable) + require.Len(t, result.Rows, 2) + + alpha := result.Rows[0] + require.Equal(t, "alpha", alpha.ModelName) + require.EqualValues(t, 2, alpha.ChannelCount) + require.EqualValues(t, 3, alpha.ObservedCount) + require.EqualValues(t, 2, alpha.ConsumeLogCount) + require.EqualValues(t, 1, alpha.ErrorLogCount) + require.EqualValues(t, 200, alpha.ConsumedQuota) + require.EqualValues(t, 1, alpha.RetryLogCount) + require.EqualValues(t, 2, alpha.LatencySampleCount) + require.EqualValues(t, 2, alpha.TotalUseTimeSeconds) + require.EqualValues(t, now-18, alpha.LastObservedAt) + + require.Equal(t, "beta", result.Rows[1].ModelName) + require.EqualValues(t, 1, result.Rows[1].ChannelCount) + require.EqualValues(t, 1, result.Rows[1].ObservedCount) + require.EqualValues(t, 50, result.Rows[1].ConsumedQuota) + require.EqualValues(t, 1, result.Rows[1].LatencySampleCount) + + require.EqualValues(t, 2, result.Summary.ModelCount) + require.EqualValues(t, 2, result.Summary.ChannelCount) + require.EqualValues(t, 4, result.Summary.ObservedCount) + require.EqualValues(t, 3, result.Summary.ConsumeLogCount) + require.EqualValues(t, 1, result.Summary.ErrorLogCount) + require.EqualValues(t, 250, result.Summary.ConsumedQuota) + require.EqualValues(t, 1, result.Summary.RetryLogCount) + require.EqualValues(t, 3, result.Summary.LatencySampleCount) + require.EqualValues(t, now-10, result.Summary.LastObservedAt) +} + +func TestQueryLegacyModelPerformanceSummaryIsNotLimitedWithRows(t *testing.T) { + db := setupChannelPerformanceTestDB(t, true) + now := int64(1_700_000_000) + logs := []Log{ + {CreatedAt: now - 1, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 1, Group: "prod", Quota: 10}, + {CreatedAt: now - 2, Type: LogTypeConsume, ModelName: "beta", ChannelId: 2, Group: "prod", Quota: 20}, + } + require.NoError(t, db.Create(&logs).Error) + + result, err := QueryLegacyModelPerformance(context.Background(), ModelPerformanceQuery{ + StartAt: now - 100, + EndAt: now, + Limit: 1, + }) + require.NoError(t, err) + require.Len(t, result.Rows, 2, "the row query should retain one extra row for truncation detection") + require.EqualValues(t, 2, result.Summary.ModelCount) + require.EqualValues(t, 2, result.Summary.ObservedCount) + require.EqualValues(t, 30, result.Summary.ConsumedQuota) +} + +func TestQueryLegacyModelPerformanceSupportsModelAndGroupFilters(t *testing.T) { + db := setupChannelPerformanceTestDB(t, true) + now := int64(1_700_000_000) + require.NoError(t, db.Create(&[]Log{ + {CreatedAt: now - 2, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 1, Group: "prod"}, + {CreatedAt: now - 1, Type: LogTypeConsume, ModelName: "alpha", ChannelId: 2, Group: "staging"}, + }).Error) + + result, err := QueryLegacyModelPerformance(context.Background(), ModelPerformanceQuery{ + StartAt: now - 100, + EndAt: now, + Limit: 10, + ModelName: "alpha", + Group: "prod", + }) + require.NoError(t, err) + require.Len(t, result.Rows, 1) + require.EqualValues(t, 1, result.Rows[0].ChannelCount) + require.EqualValues(t, 1, result.Summary.ModelCount) + require.EqualValues(t, 1, result.Summary.ChannelCount) +} diff --git a/model/payment_method_guard_test.go b/model/payment_method_guard_test.go index 3b10fc22..92bd6fff 100644 --- a/model/payment_method_guard_test.go +++ b/model/payment_method_guard_test.go @@ -1,7 +1,9 @@ package model import ( + "context" "errors" + "sync" "testing" "time" @@ -9,6 +11,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "gorm.io/gorm" + "gorm.io/gorm/schema" ) func insertUserForPaymentGuardTest(t *testing.T, id int, quota int) { @@ -333,6 +336,212 @@ func TestRechargeCreemRejectsZeroQuotaBeforeCompletingOrder(t *testing.T) { assert.EqualValues(t, 0, getUserQuotaForPaymentGuardTest(t, 610)) } +func TestRechargeCreemRejectsQuotaAboveWalletLimitWithoutCompletingOrder(t *testing.T) { + truncateTables(t) + + insertUserForPaymentGuardTest(t, 617, common.MaxQuota-2) + topUp := &TopUp{ + UserId: 617, + Amount: 2, + Money: 9.99, + TradeNo: "creem-wallet-limit", + PaymentMethod: PaymentProviderCreem, + PaymentProvider: PaymentProviderCreem, + Status: common.TopUpStatusPending, + CreateTime: time.Now().Unix(), + } + require.NoError(t, topUp.Insert()) + + err := RechargeCreem("creem-wallet-limit", "", "", "127.0.0.1") + require.ErrorIs(t, err, ErrTopUpNeedsReconciliation) + require.ErrorIs(t, RechargeCreem("creem-wallet-limit", "", "", "127.0.0.1"), ErrTopUpNeedsReconciliation) + assert.Equal(t, common.TopUpStatusPaidReconciliation, getTopUpStatusForPaymentGuardTest(t, "creem-wallet-limit")) + assert.EqualValues(t, common.MaxQuota-2, getUserQuotaForPaymentGuardTest(t, 617)) +} + +func TestRechargeCreemAllowsExactWalletBoundary(t *testing.T) { + truncateTables(t) + + insertUserForPaymentGuardTest(t, 618, common.MaxQuota-2) + topUp := &TopUp{ + UserId: 618, + Amount: 1, + Money: 9.99, + TradeNo: "creem-wallet-boundary", + PaymentMethod: PaymentProviderCreem, + PaymentProvider: PaymentProviderCreem, + Status: common.TopUpStatusPending, + CreateTime: time.Now().Unix(), + } + require.NoError(t, topUp.Insert()) + + require.NoError(t, RechargeCreem("creem-wallet-boundary", "", "", "127.0.0.1")) + assert.Equal(t, common.TopUpStatusSuccess, getTopUpStatusForPaymentGuardTest(t, "creem-wallet-boundary")) + assert.EqualValues(t, common.MaxQuota-1, getUserQuotaForPaymentGuardTest(t, 618)) +} + +func TestCreditTopUpQuotaRejectsInvalidQuotaAndMissingUser(t *testing.T) { + truncateTables(t) + insertUserForPaymentGuardTest(t, 619, 10) + + for _, creditedQuota := range []int64{0, -1, int64(common.MaxQuota)} { + err := DB.Transaction(func(tx *gorm.DB) error { + return creditTopUpQuota(tx, 619, creditedQuota, nil, nil) + }) + require.ErrorIs(t, err, ErrInvalidTopUpQuota) + } + require.EqualValues(t, 10, getUserQuotaForPaymentGuardTest(t, 619)) + + err := DB.Transaction(func(tx *gorm.DB) error { + return creditTopUpQuota(tx, 999999, 1, nil, nil) + }) + require.ErrorIs(t, err, gorm.ErrRecordNotFound) +} + +func TestRechargeCreemDuplicateCallbackCreditsExactlyOnce(t *testing.T) { + truncateTables(t) + insertUserForPaymentGuardTest(t, 620, 0) + insertTopUpForPaymentGuardTest(t, "creem-duplicate", 620, PaymentProviderCreem) + + require.NoError(t, RechargeCreem("creem-duplicate", "", "", "127.0.0.1")) + require.NoError(t, RechargeCreem("creem-duplicate", "", "", "127.0.0.1")) + + assert.Equal(t, common.TopUpStatusSuccess, getTopUpStatusForPaymentGuardTest(t, "creem-duplicate")) + assert.EqualValues(t, 2, getUserQuotaForPaymentGuardTest(t, 620)) +} + +func TestConcurrentDistinctTopUpsCannotExceedWalletLimit(t *testing.T) { + truncateTables(t) + insertUserForPaymentGuardTest(t, 621, common.MaxQuota-3) + insertTopUpForPaymentGuardTest(t, "creem-concurrent-a", 621, PaymentProviderCreem) + insertTopUpForPaymentGuardTest(t, "creem-concurrent-b", 621, PaymentProviderCreem) + + var wg sync.WaitGroup + errs := make(chan error, 2) + for _, tradeNo := range []string{"creem-concurrent-a", "creem-concurrent-b"} { + wg.Add(1) + go func(tradeNo string) { + defer wg.Done() + errs <- RechargeCreem(tradeNo, "", "", "127.0.0.1") + }(tradeNo) + } + wg.Wait() + close(errs) + + successes := 0 + failures := 0 + for err := range errs { + if err == nil { + successes++ + } else { + failures++ + } + } + assert.Equal(t, 1, successes) + assert.Equal(t, 1, failures) + assert.EqualValues(t, common.MaxQuota-1, getUserQuotaForPaymentGuardTest(t, 621)) + + statuses := []string{ + getTopUpStatusForPaymentGuardTest(t, "creem-concurrent-a"), + getTopUpStatusForPaymentGuardTest(t, "creem-concurrent-b"), + } + assert.ElementsMatch(t, []string{common.TopUpStatusSuccess, common.TopUpStatusPaidReconciliation}, statuses) +} + +func TestManualCompleteTopUpRecoversPaidReconciliationOrder(t *testing.T) { + truncateTables(t) + insertUserForPaymentGuardTest(t, 624, common.MaxQuota-2) + insertTopUpForPaymentGuardTest(t, "creem-manual-reconciliation", 624, PaymentProviderCreem) + + require.ErrorIs(t, RechargeCreem("creem-manual-reconciliation", "", "", "127.0.0.1"), ErrTopUpNeedsReconciliation) + assert.Equal(t, common.TopUpStatusPaidReconciliation, getTopUpStatusForPaymentGuardTest(t, "creem-manual-reconciliation")) + assert.EqualValues(t, common.MaxQuota-2, getUserQuotaForPaymentGuardTest(t, 624)) + + require.NoError(t, DB.Model(&User{}).Where("id = ?", 624).Update("quota", common.MaxQuota-4).Error) + require.NoError(t, ManualCompleteTopUp("creem-manual-reconciliation", "127.0.0.1")) + assert.Equal(t, common.TopUpStatusSuccess, getTopUpStatusForPaymentGuardTest(t, "creem-manual-reconciliation")) + assert.EqualValues(t, common.MaxQuota-2, getUserQuotaForPaymentGuardTest(t, 624)) + + require.NoError(t, ManualCompleteTopUp("creem-manual-reconciliation", "127.0.0.1")) + assert.EqualValues(t, common.MaxQuota-2, getUserQuotaForPaymentGuardTest(t, 624)) +} + +func TestRechargeCreemPersistsCacheInvalidationAfterCommittedCredit(t *testing.T) { + truncateTables(t) + insertUserForPaymentGuardTest(t, 622, 0) + insertTopUpForPaymentGuardTest(t, "creem-cache-outbox", 622, PaymentProviderCreem) + + client, _ := useFailingCacheMutationRedis(t, 1) + cacheUserForRetryTest(t, client, User{ + Id: 622, Username: "payment_guard_user", Group: "default", Quota: 0, Status: common.UserStatusEnabled, + }) + + require.NoError(t, RechargeCreem("creem-cache-outbox", "", "", "127.0.0.1")) + var pending int64 + require.NoError(t, DB.Model(&CacheInvalidationTask{}). + Where("kind = ? AND entity_key = ?", cacheInvalidationKindUser, "622"). + Count(&pending).Error) + require.EqualValues(t, 1, pending) + + processCacheInvalidationTasks() + requireCacheKeyDeletedEventually(t, client, getUserCacheKey(622)) + require.NoError(t, DB.Model(&CacheInvalidationTask{}). + Where("kind = ? AND entity_key = ?", cacheInvalidationKindUser, "622"). + Count(&pending).Error) + assert.Zero(t, pending) +} + +func TestRejectedTopUpDoesNotStageCacheInvalidation(t *testing.T) { + truncateTables(t) + insertUserForPaymentGuardTest(t, 623, common.MaxQuota-2) + insertTopUpForPaymentGuardTest(t, "creem-no-cache-task", 623, PaymentProviderCreem) + + client, _ := useFailingCacheMutationRedis(t, 100) + cacheUserForRetryTest(t, client, User{ + Id: 623, Username: "payment_guard_user", Group: "default", Quota: common.MaxQuota - 2, Status: common.UserStatusEnabled, + }) + + require.Error(t, RechargeCreem("creem-no-cache-task", "", "", "127.0.0.1")) + var pending int64 + require.NoError(t, DB.Model(&CacheInvalidationTask{}). + Where("kind = ? AND entity_key = ?", cacheInvalidationKindUser, "623"). + Count(&pending).Error) + assert.Zero(t, pending) + exists, err := client.Exists(context.Background(), getUserCacheKey(623)).Result() + require.NoError(t, err) + assert.EqualValues(t, 1, exists) + assert.Equal(t, common.TopUpStatusPaidReconciliation, getTopUpStatusForPaymentGuardTest(t, "creem-no-cache-task")) +} + +func TestTopUpUpdateMapColumnsExistInSchema(t *testing.T) { + testCases := []struct { + name string + model interface{} + columns []string + }{ + { + name: "top-up settlement", + model: &TopUp{}, + columns: []string{"status", "complete_time", "payment_method"}, + }, + { + name: "user credit and email claim", + model: &User{}, + columns: []string{"quota", "stripe_customer", "email", "normalized_email"}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + parsed, err := schema.Parse(tc.model, &sync.Map{}, schema.NamingStrategy{}) + require.NoError(t, err) + for _, column := range tc.columns { + require.Contains(t, parsed.DBNames, column) + } + }) + } +} + func TestRechargeRejectsPaidAmountMismatchBeforeCompletingOrder(t *testing.T) { truncateTables(t) @@ -465,6 +674,79 @@ func TestRechargeCreemSkipsDuplicateCustomerEmailBinding(t *testing.T) { assert.EqualValues(t, 1, count) } +func TestRechargeCreemDoesNotOverwriteEmailChangedAtClaimTime(t *testing.T) { + truncateTables(t) + insertUserForPaymentGuardTest(t, 625, 0) + insertTopUpForPaymentGuardTest(t, "creem-email-write-race", 625, PaymentProviderCreem) + + callbackName := "test:creem-email-write-race" + simulatedConcurrentWrite := false + var callbackErr error + require.NoError(t, DB.Callback().Update().Before("gorm:update").Register(callbackName, func(tx *gorm.DB) { + if simulatedConcurrentWrite { + return + } + fields, ok := tx.Statement.Dest.(map[string]interface{}) + if !ok { + return + } + if _, updatesEmail := fields["email"]; !updatesEmail { + return + } + simulatedConcurrentWrite = true + callbackErr = tx.Exec( + "UPDATE users SET email = ?, normalized_email = ? WHERE id = ?", + "newer@example.com", + "newer@example.com", + 625, + ).Error + })) + t.Cleanup(func() { + require.NoError(t, DB.Callback().Update().Remove(callbackName)) + }) + + require.NoError(t, RechargeCreem("creem-email-write-race", "creem@example.com", "", "127.0.0.1")) + require.True(t, simulatedConcurrentWrite) + require.NoError(t, callbackErr) + + var got User + require.NoError(t, DB.First(&got, 625).Error) + assert.Equal(t, "newer@example.com", got.Email) + assert.Equal(t, "newer@example.com", got.NormalizedEmail) + assert.EqualValues(t, 2, got.Quota) + assert.Equal(t, common.TopUpStatusSuccess, getTopUpStatusForPaymentGuardTest(t, "creem-email-write-race")) +} + +func TestRechargeCreemEmailClaimFailureDoesNotRollbackQuotaSettlement(t *testing.T) { + truncateTables(t) + insertUserForPaymentGuardTest(t, 626, 0) + insertTopUpForPaymentGuardTest(t, "creem-email-claim-failure", 626, PaymentProviderCreem) + + callbackName := "test:creem-email-claim-failure" + emailClaimErr := errors.New("forced email claim failure") + require.NoError(t, DB.Callback().Update().Before("gorm:update").Register(callbackName, func(tx *gorm.DB) { + fields, ok := tx.Statement.Dest.(map[string]interface{}) + if !ok { + return + } + if _, updatesEmail := fields["email"]; updatesEmail { + tx.AddError(emailClaimErr) + } + })) + t.Cleanup(func() { + require.NoError(t, DB.Callback().Update().Remove(callbackName)) + }) + + require.NoError(t, RechargeCreem("creem-email-claim-failure", "claim@example.com", "", "127.0.0.1")) + + var got User + require.NoError(t, DB.First(&got, 626).Error) + assert.Empty(t, got.Email) + assert.Empty(t, got.NormalizedEmail) + assert.EqualValues(t, 2, got.Quota) + assert.Equal(t, common.TopUpStatusSuccess, getTopUpStatusForPaymentGuardTest(t, "creem-email-claim-failure")) +} + func TestRefundSubscriptionPreConsume_IdempotentDoesNotDoubleRefund(t *testing.T) { truncateTables(t) diff --git a/model/perf_metric.go b/model/perf_metric.go index 1667adfb..8c470983 100644 --- a/model/perf_metric.go +++ b/model/perf_metric.go @@ -1,8 +1,10 @@ package model import ( + "context" "time" + "github.com/google/uuid" "gorm.io/gorm" "gorm.io/gorm/clause" ) @@ -22,15 +24,34 @@ type PerfMetric struct { GenerationMs int64 `json:"-" gorm:"default:0"` } +// PerfMetricFlushReceipt makes a node-local bucket commit idempotent across +// process restarts and ambiguous database responses. It is intentionally kept +// separate from PerfMetric so existing metric rows and query contracts remain +// unchanged. +type PerfMetricFlushReceipt struct { + Id int64 `gorm:"primaryKey"` + ReceiptKey string `gorm:"size:64;uniqueIndex;not null"` + ClaimToken string `gorm:"size:36;not null"` + CreatedAt int64 `gorm:"not null;index"` +} + +func (PerfMetricFlushReceipt) TableName() string { + return "perf_metric_flush_receipts" +} + func (PerfMetric) TableName() string { return "perf_metrics" } func UpsertPerfMetric(metric *PerfMetric) error { + return upsertPerfMetric(DB, metric) +} + +func upsertPerfMetric(db *gorm.DB, metric *PerfMetric) error { if metric == nil || metric.RequestCount == 0 { return nil } - return DB.Clauses(clause.OnConflict{ + return db.Clauses(clause.OnConflict{ Columns: []clause.Column{ {Name: "model_name"}, {Name: "group"}, @@ -48,12 +69,60 @@ func UpsertPerfMetric(metric *PerfMetric) error { }).Create(metric).Error } +// CommitPerfMetricBucket atomically records a durable receipt and applies the +// bucket counters once. A repeated receipt is a successful no-op, allowing a +// recovered Redis bucket to be cleaned without incrementing PerfMetric twice. +func CommitPerfMetricBucket(receiptKey string, metric *PerfMetric) (bool, error) { + if metric == nil || metric.RequestCount == 0 { + return false, nil + } + if receiptKey == "" { + return false, gorm.ErrInvalidData + } + + applied := false + err := DB.Transaction(func(tx *gorm.DB) error { + claimToken := uuid.NewString() + result := tx.Clauses(clause.OnConflict{ + Columns: []clause.Column{{Name: "receipt_key"}}, + DoNothing: true, + }).Create(&PerfMetricFlushReceipt{ + ReceiptKey: receiptKey, + ClaimToken: claimToken, + CreatedAt: time.Now().Unix(), + }) + if result.Error != nil { + return result.Error + } + var storedReceipt PerfMetricFlushReceipt + if err := tx.Select("claim_token").Where("receipt_key = ?", receiptKey).Take(&storedReceipt).Error; err != nil { + return err + } + // MySQL can report a row as affected for its no-op duplicate-key + // update when clientFoundRows is enabled. Ownership is therefore proven + // by a per-attempt claim rather than driver-specific RowsAffected values. + if storedReceipt.ClaimToken != claimToken { + return nil + } + if err := upsertPerfMetric(tx, metric); err != nil { + return err + } + applied = true + return nil + }) + return applied, err +} + func GetPerfMetrics(modelName string, group string, startTs int64, endTs int64) ([]PerfMetric, error) { + return GetPerfMetricsContext(context.Background(), modelName, group, startTs, endTs) +} + +func GetPerfMetricsContext(ctx context.Context, modelName string, group string, startTs int64, endTs int64) ([]PerfMetric, error) { var metrics []PerfMetric - query := DB.Model(&PerfMetric{}). + query := DB.WithContext(ctx).Model(&PerfMetric{}). Where("model_name = ? AND bucket_ts >= ? AND bucket_ts <= ?", modelName, startTs, endTs) if group != "" { - query = query.Where(commonGroupCol+" = ?", group) + query = query.Where(map[string]interface{}{"group": group}) } err := query.Order("bucket_ts ASC").Find(&metrics).Error return metrics, err @@ -66,18 +135,24 @@ type PerfMetricSummary struct { TotalLatencyMs int64 `json:"total_latency_ms"` OutputTokens int64 `json:"output_tokens"` GenerationMs int64 `json:"generation_ms"` + MinBucketTs int64 `json:"min_bucket_ts"` + MaxBucketTs int64 `json:"max_bucket_ts"` } func GetPerfMetricsSummaryAll(startTs int64, endTs int64, groups []string) ([]PerfMetricSummary, error) { + return GetPerfMetricsSummaryAllContext(context.Background(), startTs, endTs, groups) +} + +func GetPerfMetricsSummaryAllContext(ctx context.Context, startTs int64, endTs int64, groups []string) ([]PerfMetricSummary, error) { var summaries []PerfMetricSummary - query := DB.Model(&PerfMetric{}). - Select("model_name, SUM(request_count) as request_count, SUM(success_count) as success_count, SUM(total_latency_ms) as total_latency_ms, SUM(output_tokens) as output_tokens, SUM(generation_ms) as generation_ms"). + query := DB.WithContext(ctx).Model(&PerfMetric{}). + Select("model_name, SUM(request_count) as request_count, SUM(success_count) as success_count, SUM(total_latency_ms) as total_latency_ms, SUM(output_tokens) as output_tokens, SUM(generation_ms) as generation_ms, MIN(bucket_ts) as min_bucket_ts, MAX(bucket_ts) as max_bucket_ts"). Where("bucket_ts >= ? AND bucket_ts <= ?", startTs, endTs) if groups != nil { if len(groups) == 0 { return summaries, nil } - query = query.Where(commonGroupCol+" IN ?", groups) + query = query.Where(map[string]interface{}{"group": groups}) } err := query. Group("model_name"). @@ -93,6 +168,39 @@ func DeletePerfMetricsBefore(cutoffTs int64) error { return DB.Where("bucket_ts < ?", cutoffTs).Delete(&PerfMetric{}).Error } +func DeletePerfMetricFlushReceiptsBefore(cutoffTs int64) error { + if cutoffTs <= 0 { + return nil + } + return DB.Where("created_at < ?", cutoffTs).Delete(&PerfMetricFlushReceipt{}).Error +} + +func GetPerfMetricFlushReceiptKeys(receiptKeys []string) (map[string]struct{}, error) { + return GetPerfMetricFlushReceiptKeysContext(context.Background(), receiptKeys) +} + +func GetPerfMetricFlushReceiptKeysContext(ctx context.Context, receiptKeys []string) (map[string]struct{}, error) { + result := make(map[string]struct{}, len(receiptKeys)) + if len(receiptKeys) == 0 { + return result, nil + } + const batchSize = 200 + for start := 0; start < len(receiptKeys); start += batchSize { + end := min(start+batchSize, len(receiptKeys)) + var receipts []PerfMetricFlushReceipt + if err := DB.WithContext(ctx). + Select("receipt_key"). + Where("receipt_key IN ?", receiptKeys[start:end]). + Find(&receipts).Error; err != nil { + return nil, err + } + for _, receipt := range receipts { + result[receipt.ReceiptKey] = struct{}{} + } + } + return result, nil +} + func PerfMetricStartTime(hours int) int64 { if hours <= 0 { hours = 24 diff --git a/model/perf_metric_test.go b/model/perf_metric_test.go new file mode 100644 index 00000000..911930f8 --- /dev/null +++ b/model/perf_metric_test.go @@ -0,0 +1,43 @@ +package model + +import ( + "context" + "fmt" + "testing" + + "github.com/glebarez/sqlite" + "github.com/stretchr/testify/require" + "gorm.io/gorm" +) + +func TestGetPerfMetricFlushReceiptKeysBatchesLargeLookups(t *testing.T) { + previousDB := DB + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&PerfMetricFlushReceipt{})) + DB = db + t.Cleanup(func() { DB = previousDB }) + + receipts := make([]PerfMetricFlushReceipt, 0, 7) + requested := make([]string, 0, 1_205) + for index := 0; index < 1_205; index++ { + key := fmt.Sprintf("receipt-%04d", index) + requested = append(requested, key) + if index%200 == 0 { + receipts = append(receipts, PerfMetricFlushReceipt{ + ReceiptKey: key, + ClaimToken: fmt.Sprintf("claim-%04d", index), + CreatedAt: int64(index + 1), + }) + } + } + require.NoError(t, db.Create(&receipts).Error) + + result, err := GetPerfMetricFlushReceiptKeysContext(context.Background(), requested) + + require.NoError(t, err) + require.Len(t, result, len(receipts)) + for _, receipt := range receipts { + require.Contains(t, result, receipt.ReceiptKey) + } +} diff --git a/model/topup.go b/model/topup.go index 9c9f4bb1..16926e7d 100644 --- a/model/topup.go +++ b/model/topup.go @@ -42,11 +42,75 @@ const ( ) var ( - ErrPaymentMethodMismatch = errors.New("payment method mismatch") - ErrTopUpNotFound = errors.New("topup not found") - ErrTopUpStatusInvalid = errors.New("topup status invalid") + ErrPaymentMethodMismatch = errors.New("payment method mismatch") + ErrTopUpNotFound = errors.New("topup not found") + ErrTopUpStatusInvalid = errors.New("topup status invalid") + ErrInvalidTopUpQuota = errors.New("invalid top-up quota") + ErrTopUpQuotaLimitExceeded = errors.New("top-up quota limit exceeded") + ErrTopUpNeedsReconciliation = errors.New("paid top-up requires manual reconciliation") ) +func topUpQuotaMaxCurrent(creditedQuota int64) (int64, error) { + if creditedQuota <= 0 || creditedQuota >= int64(common.MaxQuota) { + return 0, ErrInvalidTopUpQuota + } + return int64(common.MaxQuota) - 1 - creditedQuota, nil +} + +// ValidateTopUpQuotaCapacity is the pre-payment guard. Settlement repeats the +// same invariant in creditTopUpQuota with a single conditional UPDATE. +func ValidateTopUpQuotaCapacity(userId int, creditedQuota int64) error { + maxCurrent, err := topUpQuotaMaxCurrent(creditedQuota) + if err != nil { + return err + } + var user User + if err := DB.Select("quota").Where("id = ?", userId).First(&user).Error; err != nil { + return err + } + if user.Quota > maxCurrent { + return ErrTopUpQuotaLimitExceeded + } + return nil +} + +// creditTopUpQuota atomically enforces the wallet ceiling. The predicate and +// increment must stay in the same UPDATE so concurrent callbacks cannot both +// pass a separate read/check. +func creditTopUpQuota(tx *gorm.DB, userId int, creditedQuota int64, updates map[string]interface{}, cacheTask *CacheInvalidationTask) error { + maxCurrent, err := topUpQuotaMaxCurrent(creditedQuota) + if err != nil { + return err + } + fields := make(map[string]interface{}, len(updates)+1) + for key, value := range updates { + fields[key] = value + } + fields["quota"] = gorm.Expr("quota + ?", creditedQuota) + result := tx.Model(&User{}).Where("id = ? AND quota <= ?", userId, maxCurrent).Updates(fields) + if result.Error != nil { + return result.Error + } + if result.RowsAffected == 1 { + task, err := stageUserCacheInvalidationTx(tx, userId, false) + if err != nil { + return err + } + if cacheTask != nil { + *cacheTask = task + } + return nil + } + var count int64 + if err := tx.Model(&User{}).Where("id = ?", userId).Count(&count).Error; err != nil { + return err + } + if count == 0 { + return gorm.ErrRecordNotFound + } + return ErrTopUpQuotaLimitExceeded +} + func preservePaymentValidationError(err error) error { if errors.Is(err, ErrPaymentAmountMismatch) || errors.Is(err, ErrPaymentCurrencyMismatch) || errors.Is(err, ErrPaymentMethodMismatch) { return err @@ -72,12 +136,16 @@ func updatePendingTopUpStatusTx(tx *gorm.DB, topUp *TopUp, targetStatus string) } func markPendingTopUpSuccessTx(tx *gorm.DB, topUp *TopUp) error { + return markTopUpSuccessFromStatusTx(tx, topUp, common.TopUpStatusPending) +} + +func markTopUpSuccessFromStatusTx(tx *gorm.DB, topUp *TopUp, expectedStatus string) error { if tx == nil || topUp == nil || topUp.Id == 0 { return ErrTopUpNotFound } now := common.GetTimestamp() result := tx.Model(&TopUp{}). - Where("id = ? AND status = ?", topUp.Id, common.TopUpStatusPending). + Where("id = ? AND status = ?", topUp.Id, expectedStatus). Updates(map[string]interface{}{ "status": common.TopUpStatusSuccess, "complete_time": now, @@ -93,6 +161,48 @@ func markPendingTopUpSuccessTx(tx *gorm.DB, topUp *TopUp) error { return nil } +func markPendingTopUpPaidReconciliationTx(tx *gorm.DB, topUp *TopUp) error { + if tx == nil || topUp == nil || topUp.Id == 0 { + return ErrTopUpNotFound + } + now := common.GetTimestamp() + result := tx.Model(&TopUp{}). + Where("id = ? AND status = ?", topUp.Id, common.TopUpStatusPending). + Updates(map[string]interface{}{ + "status": common.TopUpStatusPaidReconciliation, + "complete_time": now, + }) + if result.Error != nil { + return result.Error + } + if result.RowsAffected != 1 { + return ErrTopUpStatusInvalid + } + topUp.Status = common.TopUpStatusPaidReconciliation + topUp.CompleteTime = now + return nil +} + +// settlePaidTopUpTx records exactly one durable result for an externally paid +// order: either quota is credited and the order succeeds, or quota remains +// untouched and the order enters manual reconciliation. Other errors roll the +// transaction back so the provider can retry safely. +func settlePaidTopUpTx(tx *gorm.DB, topUp *TopUp, creditedQuota int64, updates map[string]interface{}, cacheTask *CacheInvalidationTask) (bool, error) { + if err := creditTopUpQuota(tx, topUp.UserId, creditedQuota, updates, cacheTask); err != nil { + if !errors.Is(err, ErrTopUpQuotaLimitExceeded) { + return false, err + } + if err := markPendingTopUpPaidReconciliationTx(tx, topUp); err != nil { + return false, err + } + return true, nil + } + if err := markPendingTopUpSuccessTx(tx, topUp); err != nil { + return false, err + } + return false, nil +} + func (topUp *TopUp) Insert() error { var err error err = DB.Create(topUp).Error @@ -165,6 +275,8 @@ func Recharge(referenceId string, customerId string, callerIp string, validation } var quota float64 + var cacheTask CacheInvalidationTask + var needsReconciliation bool topUp := &TopUp{} refCol := "`trade_no`" @@ -182,6 +294,13 @@ func Recharge(referenceId string, customerId string, callerIp string, validation return ErrPaymentMethodMismatch } + if topUp.Status == common.TopUpStatusSuccess { + return nil + } + if topUp.Status == common.TopUpStatusPaidReconciliation { + needsReconciliation = true + return nil + } if topUp.Status != common.TopUpStatusPending { return errors.New("充值订单状态错误") } @@ -190,20 +309,15 @@ func Recharge(referenceId string, customerId string, callerIp string, validation return err } - quota = topUp.Money * common.QuotaPerUnit - if quota <= 0 { + creditedQuota, quotaErr := common.QuotaFromDecimalStrict( + decimal.NewFromFloat(topUp.Money).Mul(decimal.NewFromFloat(common.QuotaPerUnit)), + ) + if quotaErr != nil || creditedQuota <= 0 { return errors.New("无效的充值额度") } - if err := markPendingTopUpSuccessTx(tx, topUp); err != nil { - return err - } - - result := tx.Model(&User{}).Where("id = ?", topUp.UserId).Updates(map[string]interface{}{"stripe_customer": customerId, "quota": gorm.Expr("quota + ?", quota)}) - if err := ensureUserUpdateMatchedTx(tx, result, topUp.UserId, errors.New("充值用户不存在")); err != nil { - return err - } - - return nil + quota = float64(creditedQuota) + needsReconciliation, err = settlePaidTopUpTx(tx, topUp, int64(creditedQuota), map[string]interface{}{"stripe_customer": customerId}, &cacheTask) + return err }) if err != nil { @@ -213,6 +327,13 @@ func Recharge(referenceId string, customerId string, callerIp string, validation } return errors.New("充值失败,请稍后重试") } + if needsReconciliation { + return ErrTopUpNeedsReconciliation + } + if quota == 0 { + return nil + } + dispatchStagedCacheInvalidation(cacheTask) RecordTopupLog(topUp.UserId, fmt.Sprintf("使用在线充值成功,充值金额: %v,支付金额:%d", logger.FormatQuota(int(quota)), topUp.Amount), callerIp, topUp.PaymentMethod, PaymentMethodStripe) @@ -233,6 +354,8 @@ func RechargeEpay(tradeNo string, actualPaymentMethod string, callerIp string, v } var quotaToAdd int64 + var cacheTask CacheInvalidationTask + var needsReconciliation bool topUp := &TopUp{} refCol := "`trade_no`" @@ -252,6 +375,10 @@ func RechargeEpay(tradeNo string, actualPaymentMethod string, callerIp string, v if topUp.Status == common.TopUpStatusSuccess { return nil } + if topUp.Status == common.TopUpStatusPaidReconciliation { + needsReconciliation = true + return nil + } if topUp.Status != common.TopUpStatusPending { return ErrTopUpStatusInvalid @@ -261,10 +388,11 @@ func RechargeEpay(tradeNo string, actualPaymentMethod string, callerIp string, v return err } - quotaToAdd = decimal.NewFromInt(topUp.Amount).Mul(decimal.NewFromFloat(common.QuotaPerUnit)).IntPart() - if quotaToAdd <= 0 { + convertedQuota, quotaErr := common.QuotaFromDecimalStrict(decimal.NewFromInt(topUp.Amount).Mul(decimal.NewFromFloat(common.QuotaPerUnit))) + if quotaErr != nil || convertedQuota <= 0 { return errors.New("invalid top-up quota") } + quotaToAdd = int64(convertedQuota) if actualPaymentMethod != "" && topUp.PaymentMethod != actualPaymentMethod { result := tx.Model(&TopUp{}). @@ -279,16 +407,8 @@ func RechargeEpay(tradeNo string, actualPaymentMethod string, callerIp string, v topUp.PaymentMethod = actualPaymentMethod } - if err := markPendingTopUpSuccessTx(tx, topUp); err != nil { - return err - } - - result := tx.Model(&User{}).Where("id = ?", topUp.UserId).Update("quota", gorm.Expr("quota + ?", quotaToAdd)) - if err := ensureUserUpdateMatchedTx(tx, result, topUp.UserId, errors.New("top-up user not found")); err != nil { - return err - } - - return nil + needsReconciliation, err = settlePaidTopUpTx(tx, topUp, quotaToAdd, nil, &cacheTask) + return err }) if err != nil { @@ -298,6 +418,10 @@ func RechargeEpay(tradeNo string, actualPaymentMethod string, callerIp string, v } return errors.New("top-up failed, please retry later") } + if needsReconciliation { + return ErrTopUpNeedsReconciliation + } + dispatchStagedCacheInvalidation(cacheTask) if quotaToAdd > 0 { RecordTopupLog(topUp.UserId, fmt.Sprintf("使用在线充值成功,充值额度: %v,支付金额:%.2f", logger.FormatQuota(quotaToAdd), topUp.Money), callerIp, topUp.PaymentMethod, PaymentProviderEpay) @@ -467,9 +591,11 @@ func ManualCompleteTopUp(tradeNo string, callerIp string) error { } var userId int - var quotaToAdd int + var quotaToAdd int64 + var cacheTask CacheInvalidationTask var payMoney float64 var paymentMethod string + var completed bool err := DB.Transaction(func(tx *gorm.DB) error { topUp := &TopUp{} @@ -483,44 +609,60 @@ func ManualCompleteTopUp(tradeNo string, callerIp string) error { return nil } - if topUp.Status != common.TopUpStatusPending { - return errors.New("订单状态不是待支付,无法补单") + if topUp.Status != common.TopUpStatusPending && topUp.Status != common.TopUpStatusPaidReconciliation { + return errors.New("订单状态不支持补单") } + expectedStatus := topUp.Status // 计算应充值额度: // - Stripe 订单:Money 代表经分组倍率换算后的美元数量,直接 * QuotaPerUnit + // - Creem 订单:Amount 已经是最终额度 // - 其他订单(如易支付):Amount 为美元数量,* QuotaPerUnit - if topUp.PaymentProvider == PaymentProviderStripe { + switch topUp.PaymentProvider { + case PaymentProviderStripe: dQuotaPerUnit := decimal.NewFromFloat(common.QuotaPerUnit) - quotaToAdd = int(decimal.NewFromFloat(topUp.Money).Mul(dQuotaPerUnit).IntPart()) - } else { + converted, quotaErr := common.QuotaFromDecimalStrict(decimal.NewFromFloat(topUp.Money).Mul(dQuotaPerUnit)) + if quotaErr != nil { + return ErrInvalidTopUpQuota + } + quotaToAdd = int64(converted) + case PaymentProviderCreem: + quotaToAdd = topUp.Amount + default: dAmount := decimal.NewFromInt(topUp.Amount) dQuotaPerUnit := decimal.NewFromFloat(common.QuotaPerUnit) - quotaToAdd = int(dAmount.Mul(dQuotaPerUnit).IntPart()) + converted, quotaErr := common.QuotaFromDecimalStrict(dAmount.Mul(dQuotaPerUnit)) + if quotaErr != nil { + return ErrInvalidTopUpQuota + } + quotaToAdd = int64(converted) } if quotaToAdd <= 0 { return errors.New("无效的充值额度") } - if err := markPendingTopUpSuccessTx(tx, topUp); err != nil { + // 增加用户额度(立即写库,保持一致性) + if err := creditTopUpQuota(tx, topUp.UserId, quotaToAdd, nil, &cacheTask); err != nil { return err } - - // 增加用户额度(立即写库,保持一致性) - result := tx.Model(&User{}).Where("id = ?", topUp.UserId).Update("quota", gorm.Expr("quota + ?", quotaToAdd)) - if err := ensureUserUpdateMatchedTx(tx, result, topUp.UserId, errors.New("充值用户不存在")); err != nil { + if err := markTopUpSuccessFromStatusTx(tx, topUp, expectedStatus); err != nil { return err } userId = topUp.UserId payMoney = topUp.Money paymentMethod = topUp.PaymentMethod + completed = true return nil }) if err != nil { return err } + if !completed { + return nil + } + dispatchStagedCacheInvalidation(cacheTask) // 事务外记录日志,避免阻塞 RecordTopupLog(userId, fmt.Sprintf("管理员补单成功,充值金额: %v,支付金额:%f", logger.FormatQuota(quotaToAdd), payMoney), callerIp, paymentMethod, "admin") @@ -532,6 +674,8 @@ func RechargeCreem(referenceId string, customerEmail string, customerName string } var quota int64 + var cacheTask CacheInvalidationTask + var needsReconciliation bool topUp := &TopUp{} refCol := "`trade_no`" @@ -539,7 +683,7 @@ func RechargeCreem(referenceId string, customerEmail string, customerName string refCol = `"trade_no"` } - err = WithNormalizedEmailWriteTx(customerEmail, func(tx *gorm.DB) error { + err = DB.Transaction(func(tx *gorm.DB) error { err := withRowLock(tx).Where(refCol+" = ?", referenceId).First(topUp).Error if err != nil { return errors.New("充值订单不存在") @@ -549,6 +693,13 @@ func RechargeCreem(referenceId string, customerEmail string, customerName string return ErrPaymentMethodMismatch } + if topUp.Status == common.TopUpStatusSuccess { + return nil + } + if topUp.Status == common.TopUpStatusPaidReconciliation { + needsReconciliation = true + return nil + } if topUp.Status != common.TopUpStatusPending { return errors.New("充值订单状态错误") } @@ -562,26 +713,8 @@ func RechargeCreem(referenceId string, customerEmail string, customerName string if quota <= 0 { return errors.New("无效的充值额度") } - if err := markPendingTopUpSuccessTx(tx, topUp); err != nil { - return err - } - - // 构建更新字段,优先使用邮箱,如果邮箱为空则使用用户名 - updateFields := map[string]interface{}{ - "quota": gorm.Expr("quota + ?", quota), - } - - // 如果有客户邮箱,尝试更新用户邮箱(仅当用户邮箱为空时) - if err := addCreemCustomerEmailUpdateIfAvailable(tx, topUp.UserId, customerEmail, updateFields); err != nil { - return err - } - - result := tx.Model(&User{}).Where("id = ?", topUp.UserId).Updates(updateFields) - if err := ensureUserUpdateMatchedTx(tx, result, topUp.UserId, errors.New("充值用户不存在")); err != nil { - return err - } - - return nil + needsReconciliation, err = settlePaidTopUpTx(tx, topUp, quota, nil, &cacheTask) + return err }) if err != nil { @@ -591,34 +724,60 @@ func RechargeCreem(referenceId string, customerEmail string, customerName string } return errors.New("充值失败,请稍后重试") } + if needsReconciliation { + bestEffortClaimCreemCustomerEmail(topUp.UserId, customerEmail, referenceId) + return ErrTopUpNeedsReconciliation + } + if quota == 0 { + return nil + } + dispatchStagedCacheInvalidation(cacheTask) + bestEffortClaimCreemCustomerEmail(topUp.UserId, customerEmail, referenceId) RecordTopupLog(topUp.UserId, fmt.Sprintf("使用Creem充值成功,充值额度: %v,支付金额:%.2f", quota, topUp.Money), callerIp, topUp.PaymentMethod, PaymentMethodCreem) return nil } -func addCreemCustomerEmailUpdateIfAvailable(tx *gorm.DB, userId int, customerEmail string, updateFields map[string]interface{}) error { - customerEmail = NormalizeEmail(customerEmail) - if customerEmail == "" { - return nil +func bestEffortClaimCreemCustomerEmail(userId int, customerEmail string, referenceId string) { + if err := claimCreemCustomerEmailIfEmpty(userId, customerEmail); err != nil { + common.SysError(fmt.Sprintf("creem email claim failed: trade_no=%s user_id=%d error=%s", referenceId, userId, err.Error())) } +} - var user User - if err := tx.Where("id = ?", userId).First(&user).Error; err != nil { - return err - } - if user.Email != "" { +func claimCreemCustomerEmailIfEmpty(userId int, customerEmail string) error { + customerEmail = NormalizeEmail(customerEmail) + if customerEmail == "" { return nil } - - if err := ensureEmailAvailableWithTx(tx, customerEmail, user.Id); err != nil { - if errors.Is(err, ErrEmailAlreadyTaken) { - return nil + var cacheTask CacheInvalidationTask + err := WithNormalizedEmailWriteTx(customerEmail, func(tx *gorm.DB) error { + if err := ensureEmailAvailableWithTx(tx, customerEmail, userId); err != nil { + if errors.Is(err, ErrEmailAlreadyTaken) { + return nil + } + return err + } + result := tx.Model(&User{}). + Where("id = ? AND (email = ? OR email IS NULL) AND (normalized_email = ? OR normalized_email IS NULL)", userId, "", ""). + Updates(map[string]interface{}{ + "email": customerEmail, + "normalized_email": customerEmail, + }) + if result.Error != nil || result.RowsAffected == 0 { + return result.Error } + task, err := stageUserCacheInvalidationTx(tx, userId, false) + if err != nil { + return err + } + cacheTask = task + return nil + }) + if err != nil { return err } - updateFields["email"] = customerEmail - updateFields["normalized_email"] = customerEmail + dispatchStagedCacheInvalidation(cacheTask) return nil } @@ -628,6 +787,8 @@ func RechargeWaffo(tradeNo string, callerIp string, validations ...PaymentValida } var quotaToAdd int + var cacheTask CacheInvalidationTask + var needsReconciliation bool topUp := &TopUp{} refCol := "`trade_no`" @@ -648,6 +809,10 @@ func RechargeWaffo(tradeNo string, callerIp string, validations ...PaymentValida if topUp.Status == common.TopUpStatusSuccess { return nil // 幂等:已成功直接返回 } + if topUp.Status == common.TopUpStatusPaidReconciliation { + needsReconciliation = true + return nil + } if topUp.Status != common.TopUpStatusPending { return errors.New("充值订单状态错误") @@ -659,21 +824,14 @@ func RechargeWaffo(tradeNo string, callerIp string, validations ...PaymentValida dAmount := decimal.NewFromInt(topUp.Amount) dQuotaPerUnit := decimal.NewFromFloat(common.QuotaPerUnit) - quotaToAdd = int(dAmount.Mul(dQuotaPerUnit).IntPart()) - if quotaToAdd <= 0 { + convertedQuota, quotaErr := common.QuotaFromDecimalStrict(dAmount.Mul(dQuotaPerUnit)) + if quotaErr != nil || convertedQuota <= 0 { return errors.New("无效的充值额度") } + quotaToAdd = convertedQuota - if err := markPendingTopUpSuccessTx(tx, topUp); err != nil { - return err - } - - result := tx.Model(&User{}).Where("id = ?", topUp.UserId).Update("quota", gorm.Expr("quota + ?", quotaToAdd)) - if err := ensureUserUpdateMatchedTx(tx, result, topUp.UserId, errors.New("充值用户不存在")); err != nil { - return err - } - - return nil + needsReconciliation, err = settlePaidTopUpTx(tx, topUp, int64(quotaToAdd), nil, &cacheTask) + return err }) if err != nil { @@ -683,6 +841,10 @@ func RechargeWaffo(tradeNo string, callerIp string, validations ...PaymentValida } return errors.New("充值失败,请稍后重试") } + if needsReconciliation { + return ErrTopUpNeedsReconciliation + } + dispatchStagedCacheInvalidation(cacheTask) if quotaToAdd > 0 { RecordTopupLog(topUp.UserId, fmt.Sprintf("Waffo充值成功,充值额度: %v,支付金额: %.2f", logger.FormatQuota(quotaToAdd), topUp.Money), callerIp, topUp.PaymentMethod, PaymentMethodWaffo) @@ -697,6 +859,8 @@ func RechargeWaffoPancake(tradeNo string, validations ...PaymentValidation) (err } var quotaToAdd int + var cacheTask CacheInvalidationTask + var needsReconciliation bool topUp := &TopUp{} refCol := "`trade_no`" @@ -717,6 +881,10 @@ func RechargeWaffoPancake(tradeNo string, validations ...PaymentValidation) (err if topUp.Status == common.TopUpStatusSuccess { return nil } + if topUp.Status == common.TopUpStatusPaidReconciliation { + needsReconciliation = true + return nil + } if topUp.Status != common.TopUpStatusPending { return errors.New("充值订单状态错误") @@ -726,21 +894,14 @@ func RechargeWaffoPancake(tradeNo string, validations ...PaymentValidation) (err return err } - quotaToAdd = int(decimal.NewFromInt(topUp.Amount).Mul(decimal.NewFromFloat(common.QuotaPerUnit)).IntPart()) - if quotaToAdd <= 0 { + convertedQuota, quotaErr := common.QuotaFromDecimalStrict(decimal.NewFromInt(topUp.Amount).Mul(decimal.NewFromFloat(common.QuotaPerUnit))) + if quotaErr != nil || convertedQuota <= 0 { return errors.New("无效的充值额度") } + quotaToAdd = convertedQuota - if err := markPendingTopUpSuccessTx(tx, topUp); err != nil { - return err - } - - result := tx.Model(&User{}).Where("id = ?", topUp.UserId).Update("quota", gorm.Expr("quota + ?", quotaToAdd)) - if err := ensureUserUpdateMatchedTx(tx, result, topUp.UserId, errors.New("充值用户不存在")); err != nil { - return err - } - - return nil + needsReconciliation, err = settlePaidTopUpTx(tx, topUp, int64(quotaToAdd), nil, &cacheTask) + return err }) if err != nil { @@ -750,6 +911,10 @@ func RechargeWaffoPancake(tradeNo string, validations ...PaymentValidation) (err } return errors.New("充值失败,请稍后重试") } + if needsReconciliation { + return ErrTopUpNeedsReconciliation + } + dispatchStagedCacheInvalidation(cacheTask) if quotaToAdd > 0 { RecordLog(topUp.UserId, LogTypeTopup, fmt.Sprintf("Waffo Pancake充值成功,充值额度: %v,支付金额: %.2f", logger.FormatQuota(quotaToAdd), topUp.Money)) diff --git a/pkg/perf_metrics/flush.go b/pkg/perf_metrics/flush.go index ae34fedd..7b31ff95 100644 --- a/pkg/perf_metrics/flush.go +++ b/pkg/perf_metrics/flush.go @@ -1,6 +1,7 @@ package perfmetrics import ( + "context" "fmt" "strconv" "time" @@ -14,60 +15,154 @@ func flushLoop() { for { interval := perf_metrics_setting.GetFlushIntervalMinutes() time.Sleep(time.Duration(interval) * time.Minute) - setting := perf_metrics_setting.GetSetting() - if !setting.Enabled { - continue - } - flushCompletedBuckets() - cleanupExpiredMetrics(setting.RetentionDays) + flushOnce() } } +func flushOnce() { + setting := perf_metrics_setting.GetSetting() + // Disabling collection stops new samples at Record, but buckets that + // were already accepted still need to be drained and committed. Skipping + // the flush here would strand those samples in memory and leave committed + // Redis snapshots around until their TTL expires. + flushCompletedBuckets() + cleanupCommittedRedisBuckets() + cleanupExpiredMetrics(setting.RetentionDays) +} + func flushCompletedBuckets() { - currentBucket := bucketStart(time.Now().Unix()) + now := time.Now().Unix() hotBuckets.Range(func(key, value any) bool { k := key.(bucketKey) - if k.bucketTs >= currentBucket { + bucketSeconds := k.bucketSeconds + if bucketSeconds <= 0 { + bucketSeconds = perf_metrics_setting.GetBucketSeconds() + } + if bucketSeconds <= 0 { + bucketSeconds = 3600 + } + if k.bucketTs+bucketSeconds > now { return true } bucket := value.(*atomicBucket) - drained := bucket.drain() + drained := bucket.closeAndDrain() + hotBuckets.CompareAndDelete(k, bucket) if drained.requestCount == 0 { - deleteOldEmptyBucket(k, key) + cleanupFlushedBucket(k) return true } - err := model.UpsertPerfMetric(&model.PerfMetric{ - ModelName: k.model, - Group: k.group, - BucketTs: k.bucketTs, - RequestCount: drained.requestCount, - SuccessCount: drained.successCount, - TotalLatencyMs: drained.totalLatencyMs, - TtftSumMs: drained.ttftSumMs, - TtftCount: drained.ttftCount, - OutputTokens: drained.outputTokens, - GenerationMs: drained.generationMs, - }) + _, err := persistBucket(k, drained) if err != nil { - bucket.addCounters(drained) + requeueCounters(k, drained) common.SysError(fmt.Sprintf("failed to flush perf metric bucket model=%s group=%s bucket=%d: %s", k.model, k.group, k.bucketTs, err.Error())) return true } - deleteOldEmptyBucket(k, key) + committedBuckets.Store(k, struct{}{}) + cleanupFlushedBucket(k) return true }) } -func deleteOldEmptyBucket(k bucketKey, rawKey any) { - if k.bucketTs < bucketStart(time.Now().Add(-24*time.Hour).Unix()) { - hotBuckets.Delete(rawKey) +func persistBucket(key bucketKey, value counters) (bool, error) { + return model.CommitPerfMetricBucket(perfMetricReceiptKey(key), &model.PerfMetric{ + ModelName: key.model, + Group: key.group, + BucketTs: key.bucketTs, + RequestCount: value.requestCount, + SuccessCount: value.successCount, + TotalLatencyMs: value.totalLatencyMs, + TtftSumMs: value.ttftSumMs, + TtftCount: value.ttftCount, + OutputTokens: value.outputTokens, + GenerationMs: value.generationMs, + }) +} + +func perfMetricReceiptKey(key bucketKey) string { + identity := fmt.Sprintf("%s\x00%s\x00%s\x00%d\x00%d", key.node, key.model, key.group, key.bucketTs, key.bucketSeconds) + return fmt.Sprintf("%x", common.Sha256Raw([]byte(identity))) +} + +func requeueCounters(key bucketKey, value counters) { + for { + actual, _ := hotBuckets.LoadOrStore(key, &atomicBucket{}) + if actual.(*atomicBucket).addCounters(value) { + return + } + hotBuckets.CompareAndDelete(key, actual) + } +} + +func cleanupFlushedBucket(key bucketKey) { + if err := deleteRedisBucket(key); err != nil { + common.SysError(fmt.Sprintf("failed to cleanup flushed perf metric bucket model=%s group=%s bucket=%d: %s", key.model, key.group, key.bucketTs, err.Error())) + return } + committedBuckets.Delete(key) + clearProjectionGap(key) + removeClosedBucket(key) +} + +func cleanupCommittedRedisBuckets() { + committedBuckets.Range(func(key, _ any) bool { + if err := deleteRedisBucket(key.(bucketKey)); err == nil { + committedBuckets.Delete(key) + clearProjectionGap(key.(bucketKey)) + removeClosedBucket(key.(bucketKey)) + } + return true + }) +} + +func removeClosedBucket(key bucketKey) { + value, ok := hotBuckets.Load(key) + if !ok { + return + } + bucket := value.(*atomicBucket) + bucket.mu.Lock() + closed := bucket.closed + bucket.mu.Unlock() + if closed { + hotBuckets.CompareAndDelete(key, bucket) + } +} + +func deleteRedisBucket(key bucketKey) error { + if !common.RedisEnabled || common.RDB == nil { + return nil + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + redisKey := redisBucketKey(key) + if err := markRedisBucketFlushed(ctx, redisKey); err != nil { + return err + } + pipe := common.RDB.TxPipeline() + pipe.ZRem(ctx, redisActiveBucketIndex, redisKey) + pipe.ZRem(ctx, redisNodeBucketIndex(key.node), redisKey) + pipe.ZRem(ctx, redisModelBucketIndex(key.model), redisKey) + pipe.Del(ctx, redisKey) + _, err := pipe.Exec(ctx) + return err +} + +func markRedisBucketFlushed(ctx context.Context, redisKey string) error { + pipe := common.RDB.TxPipeline() + pipe.HSet(ctx, redisKey, "flushed", 1) + pipe.Expire(ctx, redisKey, redisBucketTTL) + _, err := pipe.Exec(ctx) + return err } func cleanupExpiredMetrics(retentionDays int) { + receiptCutoff := time.Now().Add(-redisBucketTTL - time.Hour).Unix() + if err := model.DeletePerfMetricFlushReceiptsBefore(receiptCutoff); err != nil { + common.SysError("failed to cleanup performance metric flush receipts: " + err.Error()) + } if retentionDays <= 0 { return } diff --git a/pkg/perf_metrics/metrics.go b/pkg/perf_metrics/metrics.go index 620fddf1..fdfa1863 100644 --- a/pkg/perf_metrics/metrics.go +++ b/pkg/perf_metrics/metrics.go @@ -2,26 +2,115 @@ package perfmetrics import ( "context" + "crypto/sha256" "fmt" "math" "sort" + "strconv" "sync" + "sync/atomic" "time" "github.com/MAX-API-Next/MAX-API/common" "github.com/MAX-API-Next/MAX-API/model" relaycommon "github.com/MAX-API-Next/MAX-API/relay/common" "github.com/MAX-API-Next/MAX-API/setting/perf_metrics_setting" + "github.com/go-redis/redis/v8" + "github.com/google/uuid" ) var hotBuckets sync.Map +// committedBuckets suppresses a just-committed Redis snapshot in this +// process until cleanup succeeds. The durable flush receipt is the +// cross-restart guard; this local marker keeps the normal DB-success/ +// Redis-cleanup window from being queried twice before cleanup finishes. +var committedBuckets sync.Map + +// projectionGapBuckets tracks active local buckets whose latest absolute +// snapshot could not be published to Redis. The node health projection makes +// this quality loss visible to queries served by other instances after Redis +// recovers. +var projectionGapBuckets sync.Map +var projectionGapCount atomic.Int64 +var projectionDegradedUntil atomic.Int64 +var restoredProjectionDegraded atomic.Bool +var projectionNodeClaimConflict atomic.Bool +var projectionNodeClaimToken = uuid.NewString() + +var getPerfMetricFlushReceiptKeys = model.GetPerfMetricFlushReceiptKeysContext + +const ( + redisActiveBucketIndex = "perf:v2:active" + redisProjectionHealthIndex = "perf:v2:projection-health" + redisBucketTTL = 48 * time.Hour + projectionHealthInterval = 5 * time.Second + projectionHealthyTTL = 30 * time.Second + projectionNodeClaimTTL = 30 * time.Second +) + +var redisBucketSnapshotScript = redis.NewScript(` +redis.call('HSET', KEYS[1], + 'model', ARGV[1], + 'group', ARGV[2], + 'node', ARGV[3], + 'bucket_ts', ARGV[4], + 'bucket_seconds', ARGV[5]) +local names = {'req', 'ok', 'lat', 'ttft', 'ttft_n', 'out', 'gen_ms'} +for i = 1, #names do + local incoming = tonumber(ARGV[i + 5]) or 0 + local current = tonumber(redis.call('HGET', KEYS[1], names[i]) or '0') + if incoming > current then + redis.call('HSET', KEYS[1], names[i], incoming) + end +end +redis.call('EXPIRE', KEYS[1], tonumber(ARGV[13])) +redis.call('ZADD', KEYS[2], tonumber(ARGV[14]), KEYS[1]) +redis.call('ZADD', KEYS[3], tonumber(ARGV[14]), KEYS[1]) +redis.call('ZADD', KEYS[4], tonumber(ARGV[14]), KEYS[1]) +redis.call('EXPIRE', KEYS[2], tonumber(ARGV[15])) +redis.call('EXPIRE', KEYS[3], tonumber(ARGV[15])) +redis.call('EXPIRE', KEYS[4], tonumber(ARGV[15])) +return 1 +`) + +var redisNodeClaimScript = redis.NewScript(` +local owner = redis.call('GET', KEYS[1]) +if not owner then + redis.call('SET', KEYS[1], ARGV[1], 'EX', ARGV[2], 'NX') + owner = redis.call('GET', KEYS[1]) +end +if owner == ARGV[1] then + redis.call('EXPIRE', KEYS[1], ARGV[2]) + return 1 +end +return 0 +`) + // seriesSchema is a stable client cache/schema marker. Do not change it when // hiding fields or making response-only privacy hardening changes. const seriesSchema = "dbcd0a3c01b55203" func Init() { + if common.RedisEnabled && common.RDB != nil { + if currentNodeName() == "unnamed" { + common.SysError("performance metrics Redis recovery requires a unique stable NODE_NAME per running instance") + } + maintainProjectionNodeClaim() + } + if err := restoreProjectionHealth(); err != nil { + common.SysError("failed to restore performance projection health: " + err.Error()) + } + if recovered, err := recoverRedisBucketsForNode(); err != nil { + common.SysError("failed to recover active performance metric buckets from Redis: " + err.Error()) + } else if recovered > 0 { + common.SysLog(fmt.Sprintf("recovered %d active performance metric buckets from Redis", recovered)) + } + if err := publishProjectionHealth(); err != nil && common.RedisEnabled { + common.SysError("failed to publish initial performance projection health: " + err.Error()) + } go flushLoop() + go projectionHealthLoop() } func RecordRelaySample(info *relaycommon.RelayInfo, success bool, outputTokens int64) { @@ -66,17 +155,58 @@ func Record(sample Sample) { sample.LatencyMs = 0 } + bucketSeconds := perf_metrics_setting.GetBucketSeconds() + if bucketSeconds <= 0 { + bucketSeconds = 3600 + } + now := time.Now().Unix() key := bucketKey{ - model: sample.Model, - group: sample.Group, - bucketTs: bucketStart(time.Now().Unix()), + model: sample.Model, + group: sample.Group, + node: currentNodeName(), + bucketTs: bucketStartFor(now, bucketSeconds), + bucketSeconds: bucketSeconds, + } + projectToRedis := common.RedisEnabled && common.RDB != nil + for { + actual, _ := hotBuckets.LoadOrStore(key, &atomicBucket{}) + bucket := actual.(*atomicBucket) + if !projectToRedis { + if bucket.add(sample) { + break + } + hotBuckets.CompareAndDelete(key, actual) + continue + } + snapshot, version, added := bucket.addForProjection(sample) + if added { + success := recordRedis(key, snapshot) + setProjectionGap(key, bucket.finishProjection(version, success)) + break + } + hotBuckets.CompareAndDelete(key, actual) } - actual, _ := hotBuckets.LoadOrStore(key, &atomicBucket{}) - actual.(*atomicBucket).add(sample) - recordRedis(key, sample) } func Query(params QueryParams) (QueryResult, error) { + return QueryContext(context.Background(), params) +} + +func QueryContext(ctx context.Context, params QueryParams) (QueryResult, error) { + result, err := QueryDetailedContext(ctx, params) + if err != nil { + return QueryResult{}, err + } + return result.QueryResult, nil +} + +func QueryDetailed(params QueryParams) (DetailedQueryResult, error) { + return QueryDetailedContext(context.Background(), params) +} + +func QueryDetailedContext(ctx context.Context, params QueryParams) (DetailedQueryResult, error) { + setting := perf_metrics_setting.GetSetting() + bucketSeconds := perf_metrics_setting.GetBucketSeconds() if params.Hours <= 0 { params.Hours = 24 } @@ -87,15 +217,20 @@ func Query(params QueryParams) (QueryResult, error) { startTs := endTs - int64(params.Hours)*3600 merged := map[bucketKey]counters{} - rows, err := model.GetPerfMetrics(params.Model, params.Group, startTs, endTs) + metricStartTs := bucketStartFor(startTs, bucketSeconds) + rows, err := model.GetPerfMetricsContext(ctx, params.Model, params.Group, metricStartTs, endTs) if err != nil { - return QueryResult{}, err + return DetailedQueryResult{}, err } for _, row := range rows { + if !matchesAllowedGroup(row.Group, params.AllowedGroups) { + continue + } mergeCounters(merged, bucketKey{ - model: row.ModelName, - group: row.Group, - bucketTs: row.BucketTs, + model: row.ModelName, + group: row.Group, + bucketTs: row.BucketTs, + bucketSeconds: 0, }, counters{ requestCount: row.RequestCount, successCount: row.SuccessCount, @@ -107,19 +242,20 @@ func Query(params QueryParams) (QueryResult, error) { }) } - hotBuckets.Range(func(key, value any) bool { - k := key.(bucketKey) - if k.model != params.Model || k.bucketTs < startTs || k.bucketTs > endTs { - return true - } - if params.Group != "" && k.group != params.Group { - return true - } - mergeCounters(merged, k, value.(*atomicBucket).snapshot()) - return true + activeState := mergeActiveBuckets(ctx, merged, activeBucketFilter{ + model: params.Model, + group: params.Group, + allowedGroups: allowedGroupSet(params.AllowedGroups), + startTs: metricStartTs, + endTs: endTs, }) - return buildQueryResult(params.Model, merged), nil + queryResult := buildQueryResult(params.Model, merged) + return DetailedQueryResult{ + QueryResult: queryResult, + Coverage: buildCoverage(startTs, endTs, merged, bucketSeconds), + CollectionState: resolveCollectionState(setting.Enabled, len(queryResult.Groups) > 0, activeState), + }, nil } func QuerySummaryAll(hours int, groups []string) (SummaryAllResult, error) { @@ -131,14 +267,49 @@ func QuerySummaryAll(hours int, groups []string) (SummaryAllResult, error) { } endTs := time.Now().Unix() startTs := endTs - int64(hours)*3600 - allowedGroups := allowedGroupSet(groups) + return QuerySummaryAllRange(startTs, endTs, groups) +} - rows, err := model.GetPerfMetricsSummaryAll(startTs, endTs, groups) +// QuerySummaryAllRange aggregates bucketed model metrics whose bucket start is +// inside the caller-owned time range. Call QuerySummaryAllRangeDetailed when +// an operational client also needs the requested range, selected bucket +// coverage, bucket size, and collection state. +func QuerySummaryAllRange(startTs int64, endTs int64, groups []string) (SummaryAllResult, error) { + result, err := QuerySummaryAllRangeDetailedContext(context.Background(), startTs, endTs, groups) if err != nil { return SummaryAllResult{}, err } + return SummaryAllResult{Models: result.Models}, nil +} + +func QuerySummaryAllRangeDetailed(startTs int64, endTs int64, groups []string) (DetailedSummaryAllResult, error) { + return QuerySummaryAllRangeDetailedContext(context.Background(), startTs, endTs, groups) +} + +func QuerySummaryAllRangeDetailedContext(ctx context.Context, startTs int64, endTs int64, groups []string) (DetailedSummaryAllResult, error) { + setting := perf_metrics_setting.GetSetting() + bucketSeconds := perf_metrics_setting.GetBucketSeconds() + if endTs <= startTs { + return DetailedSummaryAllResult{ + Coverage: QueryCoverage{ + RequestedStartAt: startTs, + RequestedEndAt: endTs, + BucketSeconds: bucketSeconds, + GranularityState: CoverageGranularityUnknown, + }, + CollectionState: resolveCollectionState(setting.Enabled, false, activeCollectionHealthy), + }, nil + } + allowedGroups := allowedGroupSet(groups) + + metricStartTs := bucketStartFor(startTs, bucketSeconds) + rows, err := model.GetPerfMetricsSummaryAllContext(ctx, metricStartTs, endTs, groups) + if err != nil { + return DetailedSummaryAllResult{}, err + } totals := map[string]counters{} + coverageBuckets := map[bucketKey]struct{}{} for _, row := range rows { totals[row.ModelName] = counters{ requestCount: row.RequestCount, @@ -147,21 +318,19 @@ func QuerySummaryAll(hours int, groups []string) (SummaryAllResult, error) { outputTokens: row.OutputTokens, generationMs: row.GenerationMs, } + coverageBuckets[bucketKey{model: row.ModelName, bucketTs: row.MinBucketTs}] = struct{}{} + coverageBuckets[bucketKey{model: row.ModelName, bucketTs: row.MaxBucketTs}] = struct{}{} } - hotBuckets.Range(func(key, value any) bool { - k := key.(bucketKey) - if k.bucketTs < startTs || k.bucketTs > endTs { - return true - } - if allowedGroups != nil { - if _, ok := allowedGroups[k.group]; !ok { - return true - } - } - snap := value.(*atomicBucket).snapshot() + active := map[bucketKey]counters{} + activeState := mergeActiveBuckets(ctx, active, activeBucketFilter{ + allowedGroups: allowedGroups, + startTs: metricStartTs, + endTs: endTs, + }) + for k, snap := range active { if snap.requestCount == 0 { - return true + continue } cur := totals[k.model] cur.requestCount += snap.requestCount @@ -170,8 +339,8 @@ func QuerySummaryAll(hours int, groups []string) (SummaryAllResult, error) { cur.outputTokens += snap.outputTokens cur.generationMs += snap.generationMs totals[k.model] = cur - return true - }) + coverageBuckets[k] = struct{}{} + } models := make([]ModelSummary, 0, len(totals)) for name, total := range totals { @@ -196,7 +365,12 @@ func QuerySummaryAll(hours int, groups []string) (SummaryAllResult, error) { return models[i].RequestCount > models[j].RequestCount }) - return SummaryAllResult{Models: models}, nil + coverage := buildCoverageFromBucketSet(startTs, endTs, coverageBuckets, bucketSeconds) + return DetailedSummaryAllResult{ + Models: models, + Coverage: coverage, + CollectionState: resolveCollectionState(setting.Enabled, len(models) > 0, activeState), + }, nil } func allowedGroupSet(groups []string) map[string]struct{} { @@ -210,14 +384,29 @@ func allowedGroupSet(groups []string) map[string]struct{} { return allowed } -func bucketStart(ts int64) int64 { - bucketSeconds := perf_metrics_setting.GetBucketSeconds() +func bucketStartFor(ts int64, bucketSeconds int64) int64 { if bucketSeconds <= 0 { bucketSeconds = 3600 } return ts - (ts % bucketSeconds) } +func bucketStart(ts int64) int64 { + return bucketStartFor(ts, perf_metrics_setting.GetBucketSeconds()) +} + +func matchesAllowedGroup(group string, groups []string) bool { + if groups == nil { + return true + } + for _, allowed := range groups { + if group == allowed { + return true + } + } + return false +} + func mergeCounters(merged map[bucketKey]counters, key bucketKey, value counters) { if value.requestCount == 0 { return @@ -242,7 +431,15 @@ func buildQueryResult(modelName string, merged map[bucketKey]counters) QueryResu if _, ok := groupBuckets[key.group]; !ok { groupBuckets[key.group] = map[int64]counters{} } - groupBuckets[key.group][key.bucketTs] = value + bucket := groupBuckets[key.group][key.bucketTs] + bucket.requestCount += value.requestCount + bucket.successCount += value.successCount + bucket.totalLatencyMs += value.totalLatencyMs + bucket.ttftSumMs += value.ttftSumMs + bucket.ttftCount += value.ttftCount + bucket.outputTokens += value.outputTokens + bucket.generationMs += value.generationMs + groupBuckets[key.group][key.bucketTs] = bucket } groups := make([]string, 0, len(groupBuckets)) @@ -289,10 +486,139 @@ func buildQueryResult(modelName string, merged map[bucketKey]counters) QueryResu return QueryResult{ ModelName: modelName, SeriesSchema: seriesSchema, + Summary: buildAggregateResult(merged), Groups: results, } } +func buildAggregateResult(merged map[bucketKey]counters) AggregateResult { + total := counters{} + buckets := map[int64]counters{} + for key, value := range merged { + if value.requestCount == 0 { + continue + } + total.requestCount += value.requestCount + total.successCount += value.successCount + total.totalLatencyMs += value.totalLatencyMs + total.ttftSumMs += value.ttftSumMs + total.ttftCount += value.ttftCount + total.outputTokens += value.outputTokens + total.generationMs += value.generationMs + bucket := buckets[key.bucketTs] + bucket.requestCount += value.requestCount + bucket.successCount += value.successCount + bucket.totalLatencyMs += value.totalLatencyMs + bucket.ttftSumMs += value.ttftSumMs + bucket.ttftCount += value.ttftCount + bucket.outputTokens += value.outputTokens + bucket.generationMs += value.generationMs + buckets[key.bucketTs] = bucket + } + + timestamps := make([]int64, 0, len(buckets)) + for ts := range buckets { + timestamps = append(timestamps, ts) + } + sort.Slice(timestamps, func(i, j int) bool { return timestamps[i] < timestamps[j] }) + series := make([]BucketPoint, 0, len(timestamps)) + for _, ts := range timestamps { + series = append(series, bucketPoint(ts, buckets[ts])) + } + + return AggregateResult{ + AvgTtftMs: avg(total.ttftSumMs, total.ttftCount), + AvgLatencyMs: avg(total.totalLatencyMs, total.requestCount), + SuccessRate: successRate(total), + AvgTps: avgTps(total), + Series: series, + } +} + +func buildCoverage(startTs int64, endTs int64, merged map[bucketKey]counters, bucketSeconds int64) QueryCoverage { + buckets := make(map[bucketKey]struct{}, len(merged)) + for key, value := range merged { + if value.requestCount > 0 { + buckets[key] = struct{}{} + } + } + return buildCoverageFromBucketSet(startTs, endTs, buckets, bucketSeconds) +} + +func buildCoverageFromBucketSet(startTs int64, endTs int64, buckets map[bucketKey]struct{}, bucketSeconds int64) QueryCoverage { + if bucketSeconds <= 0 { + bucketSeconds = 3600 + } + coverage := QueryCoverage{ + RequestedStartAt: startTs, + RequestedEndAt: endTs, + BucketSeconds: bucketSeconds, + GranularityState: CoverageGranularityUnknown, + } + if len(buckets) == 0 { + return coverage + } + minTs, maxEndTs := int64(0), int64(0) + hasBucket := false + hasUnknown := false + knownGranularities := map[int64]struct{}{} + for key := range buckets { + ts := key.bucketTs + if !hasBucket || ts < minTs { + minTs = ts + } + span := key.bucketSeconds + if span <= 0 { + hasUnknown = true + span = bucketSeconds + } else { + knownGranularities[span] = struct{}{} + } + if end := ts + span; !hasBucket || end > maxEndTs { + maxEndTs = end + } + hasBucket = true + } + switch { + case hasUnknown && len(knownGranularities) == 0: + coverage.GranularityState = CoverageGranularityUnknown + case hasUnknown || len(knownGranularities) > 1: + coverage.GranularityState = CoverageGranularityMixed + default: + coverage.GranularityState = CoverageGranularityKnown + for known := range knownGranularities { + coverage.BucketSeconds = known + } + } + coverage.BucketStartAt = minTs + coverage.BucketEndAt = min(maxEndTs, endTs) + coverage.Approximate = minTs != startTs || coverage.BucketEndAt != endTs || coverage.GranularityState != CoverageGranularityKnown + return coverage +} + +type activeCollectionHealth int + +const ( + activeCollectionHealthy activeCollectionHealth = iota + activeCollectionFailed +) + +func resolveCollectionState(enabled bool, hasSamples bool, activeState activeCollectionHealth) CollectionState { + if !enabled { + return CollectionStateDisabled + } + if activeState == activeCollectionFailed && !hasSamples { + return CollectionStateQueryFailed + } + if activeState != activeCollectionHealthy && hasSamples { + return CollectionStatePartial + } + if !hasSamples { + return CollectionStateNoSamples + } + return CollectionStateAvailable +} + func bucketPoint(ts int64, value counters) BucketPoint { return BucketPoint{ Ts: ts, @@ -324,52 +650,486 @@ func avgTps(value counters) float64 { return float64(value.outputTokens) / (float64(value.generationMs) / 1000) } -func recordRedis(key bucketKey, sample Sample) { +type activeBucketFilter struct { + model string + group string + allowedGroups map[string]struct{} + startTs int64 + endTs int64 +} + +func mergeActiveBuckets(ctx context.Context, merged map[bucketKey]counters, filter activeBucketFilter) activeCollectionHealth { if !common.RedisEnabled || common.RDB == nil { + mergeMemoryBuckets(merged, &hotBuckets, filter, false) + return activeCollectionHealthy + } + + redisBuckets, err := loadRedisActiveBuckets(ctx, filter) + if err != nil { + mergeMemoryBuckets(merged, &hotBuckets, filter, false) + return activeCollectionFailed + } + for key, value := range redisBuckets { + mergeCounters(merged, key, value) + } + // The local bucket mirrors the same node-specific Redis bucket. Merge its + // monotonic counters by maximum rather than summing, so an ambiguous Redis + // write acknowledgement cannot double count the same sample. + mergeMemoryBuckets(merged, &hotBuckets, filter, true) + if localProjectionDegraded() { + return activeCollectionFailed + } + degraded, err := loadProjectionHealth(ctx) + if err != nil || degraded { + return activeCollectionFailed + } + return activeCollectionHealthy +} + +func mergeMemoryBuckets(merged map[bucketKey]counters, buckets *sync.Map, filter activeBucketFilter, mergeReplica bool) { + buckets.Range(func(key, value any) bool { + k := key.(bucketKey) + if !matchesActiveBucket(k, filter) { + return true + } + snapshot := value.(*atomicBucket).snapshot() + if mergeReplica { + mergeCounterReplica(merged, k, snapshot) + } else { + mergeCounters(merged, k, snapshot) + } + return true + }) +} + +func mergeCounterReplica(merged map[bucketKey]counters, key bucketKey, value counters) { + if value.requestCount == 0 { return } + current := merged[key] + current.requestCount = max(current.requestCount, value.requestCount) + current.successCount = max(current.successCount, value.successCount) + current.totalLatencyMs = max(current.totalLatencyMs, value.totalLatencyMs) + current.ttftSumMs = max(current.ttftSumMs, value.ttftSumMs) + current.ttftCount = max(current.ttftCount, value.ttftCount) + current.outputTokens = max(current.outputTokens, value.outputTokens) + current.generationMs = max(current.generationMs, value.generationMs) + merged[key] = current +} + +func matchesActiveBucket(key bucketKey, filter activeBucketFilter) bool { + if key.bucketTs < filter.startTs || key.bucketTs > filter.endTs { + return false + } + if filter.model != "" && key.model != filter.model { + return false + } + if filter.group != "" && key.group != filter.group { + return false + } + if filter.allowedGroups != nil { + _, ok := filter.allowedGroups[key.group] + return ok + } + return true +} + +func recordRedis(key bucketKey, value counters) bool { + if !common.RedisEnabled || common.RDB == nil { + return false + } ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() redisKey := redisBucketKey(key) - pipe := common.RDB.TxPipeline() - pipe.HIncrBy(ctx, redisKey, "req", 1) - if sample.Success { - pipe.HIncrBy(ctx, redisKey, "ok", 1) + expiresAt := time.Now().Add(redisBucketTTL).Unix() + if err := redisBucketSnapshotScript.Run(ctx, common.RDB, []string{ + redisKey, + redisActiveBucketIndex, + redisNodeBucketIndex(key.node), + redisModelBucketIndex(key.model), + }, + key.model, + key.group, + key.node, + key.bucketTs, + key.bucketSeconds, + value.requestCount, + value.successCount, + value.totalLatencyMs, + value.ttftSumMs, + value.ttftCount, + value.outputTokens, + value.generationMs, + int64(redisBucketTTL/time.Second), + expiresAt, + int64((redisBucketTTL+time.Hour)/time.Second), + ).Err(); err != nil { + return false + } + return true +} + +func loadRedisActiveBuckets(parent context.Context, filter activeBucketFilter) (map[bucketKey]counters, error) { + ctx, cancel := context.WithTimeout(parent, time.Second) + defer cancel() + index := redisActiveBucketIndex + if filter.model != "" { + index = redisModelBucketIndex(filter.model) + } + keys, err := activeRedisKeys(ctx, index) + if err != nil { + return nil, err } - if sample.LatencyMs > 0 { - pipe.HIncrBy(ctx, redisKey, "lat", sample.LatencyMs) + values, err := redisBucketValues(ctx, keys) + if err != nil { + return nil, err + } + type decodedBucket struct { + key bucketKey + value counters + receiptKey string + } + decoded := make([]decodedBucket, 0, len(values)) + receiptKeys := make([]string, 0, len(values)) + now := time.Now().Unix() + for _, fields := range values { + key, value, ok := decodeRedisBucket(fields) + if !ok || !matchesActiveBucket(key, filter) { + continue + } + if _, suppressed := committedBuckets.Load(key); suppressed { + continue + } + bucket := decodedBucket{key: key, value: value} + if key.bucketSeconds > 0 && key.bucketTs+key.bucketSeconds <= now { + bucket.receiptKey = perfMetricReceiptKey(key) + receiptKeys = append(receiptKeys, bucket.receiptKey) + } + decoded = append(decoded, bucket) } - if sample.HasTtft && sample.TtftMs >= 0 { - pipe.HIncrBy(ctx, redisKey, "ttft", sample.TtftMs) - pipe.HIncrBy(ctx, redisKey, "ttft_n", 1) + receipts := map[string]struct{}{} + if len(receiptKeys) > 0 { + var err error + receipts, err = getPerfMetricFlushReceiptKeys(ctx, receiptKeys) + if err != nil { + return nil, err + } } - if sample.OutputTokens > 0 && sample.GenerationMs > 0 { - pipe.HIncrBy(ctx, redisKey, "out", sample.OutputTokens) - pipe.HIncrBy(ctx, redisKey, "gen_ms", sample.GenerationMs) + merged := make(map[bucketKey]counters, len(decoded)) + for _, bucket := range decoded { + if bucket.receiptKey != "" { + if _, committed := receipts[bucket.receiptKey]; committed { + continue + } + } + mergeCounters(merged, bucket.key, bucket.value) } - pipe.Expire(ctx, redisKey, time.Hour) - _, _ = pipe.Exec(ctx) + return merged, nil +} + +func redisBucketKey(key bucketKey) string { + identity := sha256.Sum256([]byte(fmt.Sprintf( + "%s\x00%s\x00%s\x00%d", + key.node, + key.model, + key.group, + key.bucketSeconds, + ))) + return fmt.Sprintf("perf:v2:bucket:%x:%d", identity[:12], key.bucketTs) +} + +func redisNodeBucketIndex(node string) string { + identity := sha256.Sum256([]byte(node)) + return fmt.Sprintf("perf:v2:node:%x", identity[:12]) +} + +func redisModelBucketIndex(modelName string) string { + identity := sha256.Sum256([]byte(modelName)) + return fmt.Sprintf("perf:v2:model:%x", identity[:12]) +} + +func redisProjectionHealthKey(node string) string { + identity := sha256.Sum256([]byte(node)) + return fmt.Sprintf("perf:v2:health:%x", identity[:12]) +} + +func redisProjectionNodeClaimKey(node string) string { + identity := sha256.Sum256([]byte(node)) + return fmt.Sprintf("perf:v2:node-claim:%x", identity[:12]) } -func mergeRedisActiveBuckets(merged map[bucketKey]counters, params QueryParams, startTs int64, endTs int64) { - if !common.RedisEnabled || common.RDB == nil || params.Model == "" || params.Group == "" { +func setProjectionGap(key bucketKey, incomplete bool) { + if incomplete { + if _, loaded := projectionGapBuckets.LoadOrStore(key, struct{}{}); !loaded { + projectionGapCount.Add(1) + } + degradedUntil := time.Now().Add(redisBucketTTL).Unix() + for { + current := projectionDegradedUntil.Load() + if current >= degradedUntil || projectionDegradedUntil.CompareAndSwap(current, degradedUntil) { + break + } + } return } - active := bucketStart(time.Now().Unix()) - if active < startTs || active > endTs { + clearProjectionGap(key) +} + +func clearProjectionGap(key bucketKey) { + if _, loaded := projectionGapBuckets.LoadAndDelete(key); loaded { + if projectionGapCount.Add(-1) == 0 { + projectionDegradedUntil.Store(0) + } + } +} + +func localProjectionDegraded() bool { + return projectionGapCount.Load() > 0 || projectionDegradedUntil.Load() > time.Now().Unix() +} + +func projectionHealthLoop() { + if !common.RedisEnabled || common.RDB == nil { + return + } + ticker := time.NewTicker(projectionHealthInterval) + defer ticker.Stop() + for range ticker.C { + maintainProjectionNodeClaim() + if err := publishProjectionHealth(); err != nil && common.RedisEnabled { + common.SysError("failed to publish performance projection health: " + err.Error()) + } + } +} + +func maintainProjectionNodeClaim() { + if !common.RedisEnabled || common.RDB == nil { return } - key := bucketKey{model: params.Model, group: params.Group, bucketTs: active} ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() - values, err := common.RDB.HGetAll(ctx, redisBucketKey(key)).Result() - if err != nil || len(values) == 0 { + owned, err := claimProjectionNode(ctx, currentNodeName(), projectionNodeClaimToken) + if err != nil { + common.SysError("failed to refresh performance metrics NODE_NAME claim: " + err.Error()) + return + } + if !owned { + if projectionNodeClaimConflict.CompareAndSwap(false, true) { + common.SysError(fmt.Sprintf("duplicate NODE_NAME %q detected for performance metrics; use a unique stable NODE_NAME per running instance", currentNodeName())) + } return } - mergeCounters(merged, key, redisCounters(values)) + if projectionNodeClaimConflict.CompareAndSwap(true, false) { + common.SysLog(fmt.Sprintf("performance metrics NODE_NAME claim recovered for %q", currentNodeName())) + } } -func redisBucketKey(key bucketKey) string { - return fmt.Sprintf("perf:%s:%s:%d", key.model, key.group, key.bucketTs) +func claimProjectionNode(ctx context.Context, node string, token string) (bool, error) { + if node == "" || token == "" { + return false, fmt.Errorf("node name and claim token are required") + } + result, err := redisNodeClaimScript.Run( + ctx, + common.RDB, + []string{redisProjectionNodeClaimKey(node)}, + token, + int64(projectionNodeClaimTTL/time.Second), + ).Int() + if err != nil { + return false, err + } + return result == 1, nil +} + +func publishProjectionHealth() error { + if !common.RedisEnabled || common.RDB == nil { + return nil + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + degraded := localProjectionDegraded() + ttl := projectionHealthyTTL + if degraded { + ttl = redisBucketTTL + } + expiresAt := time.Now().Add(ttl).Unix() + key := redisProjectionHealthKey(currentNodeName()) + pipe := common.RDB.TxPipeline() + pipe.HSet(ctx, key, map[string]interface{}{ + "node": currentNodeName(), + "degraded": boolToRedisInt(degraded), + "degraded_until": projectionDegradedUntil.Load(), + "updated_at": time.Now().Unix(), + }) + pipe.Expire(ctx, key, ttl) + pipe.ZAdd(ctx, redisProjectionHealthIndex, &redis.Z{Score: float64(expiresAt), Member: key}) + pipe.Expire(ctx, redisProjectionHealthIndex, redisBucketTTL+time.Hour) + _, err := pipe.Exec(ctx) + return err +} + +func restoreProjectionHealth() error { + if !common.RedisEnabled || common.RDB == nil { + return nil + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + fields, err := common.RDB.HGetAll(ctx, redisProjectionHealthKey(currentNodeName())).Result() + if err != nil { + return err + } + degradedUntil := parseRedisInt(fields["degraded_until"]) + if fields["degraded"] == "1" && degradedUntil > time.Now().Unix() { + projectionDegradedUntil.Store(degradedUntil) + restoredProjectionDegraded.Store(true) + } + return nil +} + +func resetProjectionHealthForTest() { + projectionGapBuckets = sync.Map{} + projectionGapCount.Store(0) + projectionDegradedUntil.Store(0) + restoredProjectionDegraded.Store(false) + projectionNodeClaimConflict.Store(false) +} + +func loadProjectionHealth(parent context.Context) (bool, error) { + if !common.RedisEnabled || common.RDB == nil { + return false, nil + } + ctx, cancel := context.WithTimeout(parent, time.Second) + defer cancel() + keys, err := activeRedisKeys(ctx, redisProjectionHealthIndex) + if err != nil { + return false, err + } + values, err := redisBucketValues(ctx, keys) + if err != nil { + return false, err + } + now := time.Now().Unix() + for _, fields := range values { + if fields["degraded"] == "1" && parseRedisInt(fields["degraded_until"]) > now { + return true, nil + } + } + return false, nil +} + +func boolToRedisInt(value bool) int { + if value { + return 1 + } + return 0 +} + +func currentNodeName() string { + if common.NodeName != "" { + return common.NodeName + } + return "unnamed" +} + +func activeRedisKeys(ctx context.Context, index string) ([]string, error) { + now := time.Now().Unix() + pipe := common.RDB.TxPipeline() + pipe.ZRemRangeByScore(ctx, index, "-inf", strconv.FormatInt(now, 10)) + keysCmd := pipe.ZRangeByScore(ctx, index, &redis.ZRangeBy{Min: strconv.FormatInt(now+1, 10), Max: "+inf"}) + if _, err := pipe.Exec(ctx); err != nil { + return nil, err + } + return keysCmd.Val(), nil +} + +func redisBucketValues(ctx context.Context, keys []string) ([]map[string]string, error) { + if len(keys) == 0 { + return nil, nil + } + pipe := common.RDB.Pipeline() + commands := make([]*redis.StringStringMapCmd, 0, len(keys)) + for _, key := range keys { + commands = append(commands, pipe.HGetAll(ctx, key)) + } + if _, err := pipe.Exec(ctx); err != nil { + return nil, err + } + values := make([]map[string]string, 0, len(commands)) + for _, command := range commands { + if fields := command.Val(); len(fields) > 0 && fields["flushed"] != "1" { + values = append(values, fields) + } + } + return values, nil +} + +func decodeRedisBucket(fields map[string]string) (bucketKey, counters, bool) { + key := bucketKey{ + model: fields["model"], + group: fields["group"], + node: fields["node"], + bucketTs: parseRedisInt(fields["bucket_ts"]), + bucketSeconds: parseRedisInt(fields["bucket_seconds"]), + } + value := redisCounters(fields) + return key, value, key.model != "" && key.group != "" && key.node != "" && key.bucketTs > 0 && value.requestCount > 0 +} + +func recoverRedisBucketsForNode() (int, error) { + if !common.RedisEnabled || common.RDB == nil { + return 0, nil + } + node := currentNodeName() + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + keys, err := activeRedisKeys(ctx, redisNodeBucketIndex(node)) + if err != nil { + return 0, err + } + values, err := redisBucketValues(ctx, keys) + if err != nil { + return 0, err + } + type recoveredBucket struct { + key bucketKey + value counters + } + candidates := make([]recoveredBucket, 0, len(values)) + receiptKeys := make([]string, 0, len(values)) + for _, fields := range values { + key, value, ok := decodeRedisBucket(fields) + if !ok || key.node != node { + continue + } + candidates = append(candidates, recoveredBucket{key: key, value: value}) + receiptKeys = append(receiptKeys, perfMetricReceiptKey(key)) + } + receipts, err := getPerfMetricFlushReceiptKeys(ctx, receiptKeys) + if err != nil { + return 0, err + } + restoreGapOwnership := restoredProjectionDegraded.Swap(false) + recovered := 0 + for _, candidate := range candidates { + if _, committed := receipts[perfMetricReceiptKey(candidate.key)]; committed { + if err := deleteRedisBucket(candidate.key); err != nil { + committedBuckets.Store(candidate.key, struct{}{}) + } + continue + } + bucket := &atomicBucket{} + if !bucket.addCounters(candidate.value) { + common.SysError(fmt.Sprintf("failed to restore performance metric bucket model=%s group=%s bucket=%d", candidate.key.model, candidate.key.group, candidate.key.bucketTs)) + continue + } + hotBuckets.Store(candidate.key, bucket) + if restoreGapOwnership { + setProjectionGap(candidate.key, true) + } + recovered++ + } + if restoreGapOwnership && recovered == 0 { + projectionDegradedUntil.Store(0) + } + return recovered, nil } diff --git a/pkg/perf_metrics/metrics_test.go b/pkg/perf_metrics/metrics_test.go new file mode 100644 index 00000000..ed00caf4 --- /dev/null +++ b/pkg/perf_metrics/metrics_test.go @@ -0,0 +1,910 @@ +package perfmetrics + +import ( + "context" + "sync" + "testing" + "time" + + "github.com/MAX-API-Next/MAX-API/common" + "github.com/MAX-API-Next/MAX-API/model" + "github.com/MAX-API-Next/MAX-API/setting/config" + "github.com/MAX-API-Next/MAX-API/setting/perf_metrics_setting" + "github.com/alicebob/miniredis/v2" + "github.com/glebarez/sqlite" + "github.com/go-redis/redis/v8" + "github.com/stretchr/testify/require" + "gorm.io/gorm" +) + +func TestQuerySummaryAllRangeReportsBucketCoverageAndGroupFilter(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + model.DB = db + common.RDB = nil + common.RedisEnabled = false + hotBuckets = sync.Map{} + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + hotBuckets = sync.Map{} + }) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + require.NoError(t, db.Create(&[]model.PerfMetric{ + {ModelName: "alpha", Group: "prod", BucketTs: 1_000, RequestCount: 2, SuccessCount: 2, TotalLatencyMs: 400, OutputTokens: 100, GenerationMs: 2_000}, + {ModelName: "alpha", Group: "prod", BucketTs: 2_000, RequestCount: 1, SuccessCount: 0, TotalLatencyMs: 300, OutputTokens: 50, GenerationMs: 1_000}, + {ModelName: "alpha", Group: "staging", BucketTs: 1_500, RequestCount: 10, SuccessCount: 10, TotalLatencyMs: 100, OutputTokens: 1_000, GenerationMs: 1_000}, + {ModelName: "beta", Group: "prod", BucketTs: 2_600, RequestCount: 1, SuccessCount: 1, TotalLatencyMs: 50, OutputTokens: 100, GenerationMs: 1_000}, + }).Error) + + result, err := QuerySummaryAllRangeDetailed(900, 2_500, []string{"prod"}) + require.NoError(t, err) + require.Len(t, result.Models, 1) + require.Equal(t, "alpha", result.Models[0].ModelName) + require.EqualValues(t, 3, result.Models[0].RequestCount) + require.EqualValues(t, 233, result.Models[0].AvgLatencyMs) + require.InDelta(t, 66.67, result.Models[0].SuccessRate, 0.001) + require.InDelta(t, 50, result.Models[0].AvgTps, 0.001) + require.Equal(t, CollectionStateAvailable, result.CollectionState) + require.EqualValues(t, 900, result.Coverage.RequestedStartAt) + require.EqualValues(t, 2_500, result.Coverage.RequestedEndAt) + require.EqualValues(t, 1_000, result.Coverage.BucketStartAt) + require.EqualValues(t, 2_500, result.Coverage.BucketEndAt) + require.EqualValues(t, 3_600, result.Coverage.BucketSeconds) + require.True(t, result.Coverage.Approximate) + require.Equal(t, CoverageGranularityUnknown, result.Coverage.GranularityState) +} + +func TestBuildQueryResultUsesRawCountersForModelAggregate(t *testing.T) { + result := buildQueryResult("alpha", map[bucketKey]counters{ + {model: "alpha", group: "small", bucketTs: 3_600}: { + requestCount: 1, + successCount: 0, + totalLatencyMs: 1_000, + ttftSumMs: 100, + ttftCount: 1, + outputTokens: 10, + generationMs: 1_000, + }, + {model: "alpha", group: "large", bucketTs: 3_600}: { + requestCount: 99, + successCount: 99, + totalLatencyMs: 9_900, + ttftSumMs: 4_950, + ttftCount: 99, + outputTokens: 990, + generationMs: 9_900, + }, + }) + + require.Len(t, result.Groups, 2) + require.EqualValues(t, 109, result.Summary.AvgLatencyMs) + require.EqualValues(t, 50, result.Summary.AvgTtftMs) + require.InDelta(t, 99, result.Summary.SuccessRate, 0.001) + require.InDelta(t, 91.743, result.Summary.AvgTps, 0.001) + require.Len(t, result.Summary.Series, 1) + require.InDelta(t, 99, result.Summary.Series[0].SuccessRate, 0.001) +} + +func TestBuildQueryResultMergesSameGroupBucketAcrossNodes(t *testing.T) { + result := buildQueryResult("alpha", map[bucketKey]counters{ + {model: "alpha", group: "prod", node: "node-a", bucketTs: 3_600}: { + requestCount: 2, + successCount: 2, + totalLatencyMs: 200, + ttftSumMs: 80, + ttftCount: 2, + outputTokens: 20, + generationMs: 1_000, + }, + {model: "alpha", group: "prod", node: "node-b", bucketTs: 3_600}: { + requestCount: 3, + successCount: 2, + totalLatencyMs: 600, + ttftSumMs: 120, + ttftCount: 3, + outputTokens: 30, + generationMs: 1_500, + }, + }) + + require.Len(t, result.Groups, 1) + require.EqualValues(t, 160, result.Groups[0].AvgLatencyMs) + require.EqualValues(t, 40, result.Groups[0].AvgTtftMs) + require.InDelta(t, 80, result.Groups[0].SuccessRate, 0.001) + require.InDelta(t, 80, result.Summary.SuccessRate, 0.001) +} + +func TestAtomicBucketCloseAndDrainRejectsLateSample(t *testing.T) { + bucket := &atomicBucket{} + require.True(t, bucket.add(Sample{Success: true, LatencyMs: 100})) + drained := bucket.closeAndDrain() + require.EqualValues(t, 1, drained.requestCount) + require.False(t, bucket.add(Sample{Success: true, LatencyMs: 50})) + require.Zero(t, bucket.snapshot().requestCount) +} + +func TestFlushOnceDrainsAcceptedBucketsWhenCollectionIsDisabled(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + setting := config.GlobalConfig.Get("perf_metrics_setting").(*perf_metrics_setting.PerfMetricsSetting) + previousSetting := *setting + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + model.DB = db + common.RDB = nil + common.RedisEnabled = false + *setting = previousSetting + setting.Enabled = false + hotBuckets = sync.Map{} + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + *setting = previousSetting + hotBuckets = sync.Map{} + }) + + key := bucketKey{ + model: "alpha", + group: "prod", + node: "node-a", + bucketTs: bucketStart(time.Now().Unix()) - 3_600, + bucketSeconds: 3_600, + } + bucket := &atomicBucket{} + require.True(t, bucket.add(Sample{Success: true, LatencyMs: 125})) + hotBuckets.Store(key, bucket) + + flushOnce() + + rows, err := model.GetPerfMetrics(key.model, key.group, key.bucketTs, key.bucketTs) + require.NoError(t, err) + require.Len(t, rows, 1) + require.EqualValues(t, 1, rows[0].RequestCount) +} + +func TestAtomicBucketSnapshotWaitsForLifecycleLock(t *testing.T) { + bucket := &atomicBucket{} + bucket.mu.Lock() + done := make(chan struct{}) + go func() { + _ = bucket.snapshot() + close(done) + }() + + select { + case <-done: + bucket.mu.Unlock() + t.Fatal("snapshot returned while the bucket lifecycle lock was held") + case <-time.After(50 * time.Millisecond): + bucket.mu.Unlock() + } + + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("snapshot did not resume after the lifecycle lock was released") + } +} + +func TestAtomicBucketCloseWaitsForInFlightRedisProjection(t *testing.T) { + bucket := &atomicBucket{} + value, version, added := bucket.addForProjection(Sample{Success: true, LatencyMs: 100}) + require.True(t, added) + require.EqualValues(t, 1, value.requestCount) + + drained := make(chan counters, 1) + go func() { + drained <- bucket.closeAndDrain() + }() + select { + case <-drained: + t.Fatal("bucket drained before its in-flight Redis projection completed") + case <-time.After(50 * time.Millisecond): + } + + require.False(t, bucket.finishProjection(version, true)) + select { + case result := <-drained: + require.EqualValues(t, 1, result.requestCount) + require.EqualValues(t, 1, result.successCount) + case <-time.After(time.Second): + t.Fatal("bucket did not drain after the Redis projection completed") + } +} + +func TestRemoveClosedBucketDoesNotDeleteReplacement(t *testing.T) { + hotBuckets = sync.Map{} + key := bucketKey{model: "alpha", group: "prod", node: "node", bucketTs: 3_600, bucketSeconds: 3_600} + closed := &atomicBucket{} + require.True(t, closed.add(Sample{Success: true})) + closed.closeAndDrain() + replacement := &atomicBucket{} + require.True(t, replacement.add(Sample{Success: true})) + hotBuckets.Store(key, replacement) + t.Cleanup(func() { hotBuckets = sync.Map{} }) + + removeClosedBucket(key) + + actual, ok := hotBuckets.Load(key) + require.True(t, ok) + require.Same(t, replacement, actual) +} + +func TestQueryDetailedExcludesInactiveGroupsBeforeAggregate(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + model.DB = db + common.RDB = nil + common.RedisEnabled = false + hotBuckets = sync.Map{} + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + hotBuckets = sync.Map{} + }) + + bucketSeconds := int64(3_600) + bucketTs := bucketStartFor(time.Now().Unix(), bucketSeconds) + require.NoError(t, db.Create(&[]model.PerfMetric{ + {ModelName: "alpha", Group: "prod", BucketTs: bucketTs, RequestCount: 99, SuccessCount: 99, TotalLatencyMs: 9_900}, + {ModelName: "alpha", Group: "removed", BucketTs: bucketTs, RequestCount: 1, SuccessCount: 0, TotalLatencyMs: 1_000}, + }).Error) + + result, err := QueryDetailed(QueryParams{ + Model: "alpha", + Hours: 1, + AllowedGroups: []string{"prod"}, + }) + + require.NoError(t, err) + require.Len(t, result.Groups, 1) + require.Equal(t, "prod", result.Groups[0].Group) + require.InDelta(t, 100, result.Summary.SuccessRate, 0.001) + require.EqualValues(t, 100, result.Summary.AvgLatencyMs) +} + +func TestQueryDetailedMarksRedisFailureAsPartialWhenDatabaseHasSamples(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + require.NoError(t, redisClient.Close()) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + hotBuckets = sync.Map{} + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + hotBuckets = sync.Map{} + }) + + bucketTs := bucketStart(time.Now().Unix()) + require.NoError(t, db.Create(&model.PerfMetric{ + ModelName: "alpha", Group: "prod", BucketTs: bucketTs, + RequestCount: 1, SuccessCount: 1, TotalLatencyMs: 100, + }).Error) + + result, err := QueryDetailed(QueryParams{Model: "alpha", Hours: 1}) + + require.NoError(t, err) + require.Equal(t, CollectionStatePartial, result.CollectionState) + require.Len(t, result.Groups, 1) +} + +func TestQueryDetailedMarksRedisFailureAsQueryFailedWithoutSamples(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + require.NoError(t, redisClient.Close()) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + hotBuckets = sync.Map{} + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + hotBuckets = sync.Map{} + }) + + result, err := QueryDetailed(QueryParams{Model: "alpha", Hours: 1}) + + require.NoError(t, err) + require.Equal(t, CollectionStateQueryFailed, result.CollectionState) + require.Empty(t, result.Groups) +} + +func TestQuerySummaryAllRangeDetailedMarksEmptyWindowAsNoSamples(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + model.DB = db + common.RDB = nil + common.RedisEnabled = false + hotBuckets = sync.Map{} + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + hotBuckets = sync.Map{} + }) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + + result, err := QuerySummaryAllRangeDetailed(900, 2_500, nil) + + require.NoError(t, err) + require.Empty(t, result.Models) + require.Equal(t, CollectionStateNoSamples, result.CollectionState) + require.EqualValues(t, 900, result.Coverage.RequestedStartAt) + require.EqualValues(t, 2_500, result.Coverage.RequestedEndAt) + require.Zero(t, result.Coverage.BucketStartAt) + require.Zero(t, result.Coverage.BucketEndAt) + require.False(t, result.Coverage.Approximate) +} + +func TestResolveCollectionStateDistinguishesNormalOperationalStates(t *testing.T) { + require.Equal(t, CollectionStateDisabled, resolveCollectionState(false, true, activeCollectionHealthy)) + require.Equal(t, CollectionStateDisabled, resolveCollectionState(false, false, activeCollectionHealthy)) + require.Equal(t, CollectionStateNoSamples, resolveCollectionState(true, false, activeCollectionHealthy)) + require.Equal(t, CollectionStateAvailable, resolveCollectionState(true, true, activeCollectionHealthy)) + require.Equal(t, CollectionStatePartial, resolveCollectionState(true, true, activeCollectionFailed)) + require.Equal(t, CollectionStateQueryFailed, resolveCollectionState(true, false, activeCollectionFailed)) +} + +func TestQuerySummaryAllRangeDetailedMergesRedisBucketsAcrossNodesWithoutLocalDoubleCounting(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + previousNodeName := common.NodeName + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + hotBuckets = sync.Map{} + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + common.NodeName = previousNodeName + hotBuckets = sync.Map{} + _ = redisClient.Close() + }) + + common.NodeName = "perf-node-a" + Record(Sample{Model: "alpha", Group: "prod", LatencyMs: 100, Success: true}) + common.NodeName = "perf-node-b" + Record(Sample{Model: "alpha", Group: "prod", LatencyMs: 300, Success: false}) + + now := time.Now().Unix() + result, err := QuerySummaryAllRangeDetailed(now-3600, now+1, []string{"prod"}) + + require.NoError(t, err) + require.Len(t, result.Models, 1) + require.EqualValues(t, 2, result.Models[0].RequestCount) + require.EqualValues(t, 200, result.Models[0].AvgLatencyMs) + require.InDelta(t, 50, result.Models[0].SuccessRate, 0.001) +} + +func TestRecoverRedisBucketsForNodeRestoresFlushStateAfterRestart(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + previousNodeName := common.NodeName + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + common.NodeName = "stable-perf-node" + hotBuckets = sync.Map{} + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + common.NodeName = previousNodeName + hotBuckets = sync.Map{} + _ = redisClient.Close() + }) + + Record(Sample{Model: "alpha", Group: "prod", LatencyMs: 125, Success: true}) + hotBuckets = sync.Map{} + + recovered, err := recoverRedisBucketsForNode() + + require.NoError(t, err) + require.Equal(t, 1, recovered) + requestCount := int64(0) + hotBuckets.Range(func(_, value any) bool { + requestCount += value.(*atomicBucket).snapshot().requestCount + return true + }) + require.EqualValues(t, 1, requestCount) +} + +func TestRecoverRedisBucketsForNodeUsesContextAwareReceiptLookup(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + previousNodeName := common.NodeName + previousLookup := getPerfMetricFlushReceiptKeys + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + common.NodeName = "stable-perf-node" + hotBuckets = sync.Map{} + lookupCalls := 0 + lookupHadDeadline := false + getPerfMetricFlushReceiptKeys = func(ctx context.Context, _ []string) (map[string]struct{}, error) { + lookupCalls++ + _, lookupHadDeadline = ctx.Deadline() + return map[string]struct{}{}, nil + } + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + common.NodeName = previousNodeName + getPerfMetricFlushReceiptKeys = previousLookup + hotBuckets = sync.Map{} + _ = redisClient.Close() + }) + + Record(Sample{Model: "alpha", Group: "prod", LatencyMs: 125, Success: true}) + hotBuckets = sync.Map{} + + recovered, err := recoverRedisBucketsForNode() + + require.NoError(t, err) + require.Equal(t, 1, recovered) + require.Equal(t, 1, lookupCalls) + require.True(t, lookupHadDeadline) +} + +func TestRecoveredCompletedRedisBucketFlushesToDatabaseAndCleansActiveState(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + previousNodeName := common.NodeName + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + common.NodeName = "stable-flush-node" + hotBuckets = sync.Map{} + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + common.NodeName = previousNodeName + hotBuckets = sync.Map{} + _ = redisClient.Close() + }) + + bucketSeconds := int64(3600) + key := bucketKey{ + model: "alpha", + group: "prod", + node: common.NodeName, + bucketTs: bucketStart(time.Now().Unix()) - bucketSeconds, + bucketSeconds: bucketSeconds, + } + require.True(t, recordRedis(key, counters{ + requestCount: 1, + successCount: 1, + totalLatencyMs: 250, + })) + recovered, err := recoverRedisBucketsForNode() + require.NoError(t, err) + require.Equal(t, 1, recovered) + + flushCompletedBuckets() + + rows, err := model.GetPerfMetrics(key.model, key.group, key.bucketTs, key.bucketTs) + require.NoError(t, err) + require.Len(t, rows, 1) + require.EqualValues(t, 1, rows[0].RequestCount) + require.EqualValues(t, 1, rows[0].SuccessCount) + require.EqualValues(t, 250, rows[0].TotalLatencyMs) + require.EqualValues(t, 0, redisClient.Exists(context.Background(), redisBucketKey(key)).Val()) + _, hot := hotBuckets.Load(key) + require.False(t, hot) +} + +func TestRecoveredCommittedRedisBucketDoesNotIncrementDatabaseTwice(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + previousNodeName := common.NodeName + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + common.NodeName = "stable-idempotent-node" + hotBuckets = sync.Map{} + committedBuckets = sync.Map{} + resetProjectionHealthForTest() + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + common.NodeName = previousNodeName + hotBuckets = sync.Map{} + committedBuckets = sync.Map{} + resetProjectionHealthForTest() + _ = redisClient.Close() + }) + + key := bucketKey{ + model: "alpha", + group: "prod", + node: common.NodeName, + bucketTs: bucketStart(time.Now().Unix()) - 3600, + bucketSeconds: 3600, + } + value := counters{requestCount: 1, successCount: 1, totalLatencyMs: 100} + require.True(t, recordRedis(key, value)) + applied, err := persistBucket(key, value) + require.NoError(t, err) + require.True(t, applied) + reapplied, err := persistBucket(key, value) + require.NoError(t, err) + require.False(t, reapplied, "the durable receipt must make a repeated commit a no-op") + + // Simulate termination after the database transaction commits but before + // the Redis bucket and process-local receipt cache are cleaned. + hotBuckets = sync.Map{} + committedBuckets = sync.Map{} + recovered, err := recoverRedisBucketsForNode() + require.NoError(t, err) + require.Zero(t, recovered) + require.EqualValues(t, 0, redisClient.Exists(context.Background(), redisBucketKey(key)).Val()) + + flushCompletedBuckets() + + rows, err := model.GetPerfMetrics(key.model, key.group, key.bucketTs, key.bucketTs) + require.NoError(t, err) + require.Len(t, rows, 1) + require.EqualValues(t, 1, rows[0].RequestCount) + require.EqualValues(t, 1, rows[0].SuccessCount) + require.EqualValues(t, 100, rows[0].TotalLatencyMs) +} + +func TestRedisProjectionGapIsVisibleAcrossNodesAfterRedisRecovers(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + previousNodeName := common.NodeName + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + common.NodeName = "projection-node-a" + hotBuckets = sync.Map{} + committedBuckets = sync.Map{} + resetProjectionHealthForTest() + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + common.NodeName = previousNodeName + hotBuckets = sync.Map{} + committedBuckets = sync.Map{} + resetProjectionHealthForTest() + _ = redisClient.Close() + }) + + redisServer.Close() + Record(Sample{Model: "alpha", Group: "prod", Success: true, LatencyMs: 100}) + hotBuckets = sync.Map{} // another process cannot access node A's local bucket + require.NoError(t, redisServer.Restart()) + require.NoError(t, publishProjectionHealth()) + resetProjectionHealthForTest() // simulate the independent node B process + + common.NodeName = "projection-node-b" + Record(Sample{Model: "alpha", Group: "prod", Success: true, LatencyMs: 200}) + result, err := QueryDetailed(QueryParams{Model: "alpha", Hours: 1}) + require.NoError(t, err) + require.Equal(t, CollectionStatePartial, result.CollectionState) +} + +func TestRedisAbsoluteSnapshotIsIdempotentWhenAWriteIsRetried(t *testing.T) { + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + common.RDB = redisClient + common.RedisEnabled = true + t.Cleanup(func() { + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + _ = redisClient.Close() + }) + + key := bucketKey{model: "alpha", group: "prod", node: "node-a", bucketTs: bucketStart(time.Now().Unix()), bucketSeconds: 3600} + value := counters{requestCount: 1, successCount: 1, totalLatencyMs: 100} + require.True(t, recordRedis(key, value)) + require.True(t, recordRedis(key, value)) + + fields, err := redisClient.HGetAll(context.Background(), redisBucketKey(key)).Result() + require.NoError(t, err) + require.EqualValues(t, 1, parseRedisInt(fields["req"])) + require.EqualValues(t, 1, parseRedisInt(fields["ok"])) + require.EqualValues(t, 100, parseRedisInt(fields["lat"])) +} + +func TestModelQueryUsesModelScopedRedisIndex(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + _ = redisClient.Close() + }) + + now := time.Now().Unix() + alphaKey := bucketKey{model: "alpha", group: "prod", node: "node-a", bucketTs: bucketStart(now), bucketSeconds: 3600} + betaKey := bucketKey{model: "beta", group: "prod", node: "node-b", bucketTs: bucketStart(now), bucketSeconds: 3600} + require.True(t, recordRedis(alphaKey, counters{requestCount: 1, successCount: 1})) + require.True(t, recordRedis(betaKey, counters{requestCount: 1, successCount: 1})) + require.NoError(t, redisClient.Del(context.Background(), redisActiveBucketIndex).Err()) + + values, err := loadRedisActiveBuckets(context.Background(), activeBucketFilter{ + model: "alpha", + startTs: alphaKey.bucketTs, + endTs: now, + }) + require.NoError(t, err) + require.Len(t, values, 1) + require.Contains(t, values, alphaKey) +} + +func TestLoadRedisActiveBucketsSkipsReceiptLookupForOpenBuckets(t *testing.T) { + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + previousLookup := getPerfMetricFlushReceiptKeys + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + common.RDB = redisClient + common.RedisEnabled = true + lookupCalls := 0 + getPerfMetricFlushReceiptKeys = func(context.Context, []string) (map[string]struct{}, error) { + lookupCalls++ + return map[string]struct{}{}, nil + } + t.Cleanup(func() { + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + getPerfMetricFlushReceiptKeys = previousLookup + _ = redisClient.Close() + }) + + now := time.Now().Unix() + key := bucketKey{ + model: "alpha", + group: "prod", + node: "node-a", + bucketTs: bucketStartFor(now, 3_600), + bucketSeconds: 3_600, + } + require.True(t, recordRedis(key, counters{requestCount: 1, successCount: 1})) + + values, err := loadRedisActiveBuckets(context.Background(), activeBucketFilter{ + model: "alpha", + startTs: key.bucketTs, + endTs: now, + }) + + require.NoError(t, err) + require.Contains(t, values, key) + require.Zero(t, lookupCalls) +} + +func TestClaimProjectionNodeDetectsDuplicateOwnerAndRefreshesTTL(t *testing.T) { + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + common.RDB = redisClient + common.RedisEnabled = true + t.Cleanup(func() { + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + _ = redisClient.Close() + }) + + ctx := context.Background() + owned, err := claimProjectionNode(ctx, "shared-node", "process-a") + require.NoError(t, err) + require.True(t, owned) + owned, err = claimProjectionNode(ctx, "shared-node", "process-b") + require.NoError(t, err) + require.False(t, owned) + owned, err = claimProjectionNode(ctx, "shared-node", "process-a") + require.NoError(t, err) + require.True(t, owned) + require.Greater(t, redisClient.TTL(ctx, redisProjectionNodeClaimKey("shared-node")).Val(), time.Duration(0)) +} + +func TestMarkRedisBucketFlushedPreservesExpiryWhenHashIsMissing(t *testing.T) { + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + common.RDB = redisClient + common.RedisEnabled = true + t.Cleanup(func() { + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + _ = redisClient.Close() + }) + + ctx := context.Background() + require.NoError(t, markRedisBucketFlushed(ctx, "perf:v2:bucket:missing")) + require.Equal(t, "1", redisClient.HGet(ctx, "perf:v2:bucket:missing", "flushed").Val()) + require.Greater(t, redisClient.TTL(ctx, "perf:v2:bucket:missing").Val(), time.Duration(0)) +} + +func TestRecoveredProjectionHealthClearsWhenNoBucketCanOwnTheGap(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + previousNodeName := common.NodeName + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + model.DB = db + common.RDB = redisClient + common.RedisEnabled = true + common.NodeName = "restored-health-node" + resetProjectionHealthForTest() + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + common.NodeName = previousNodeName + resetProjectionHealthForTest() + _ = redisClient.Close() + }) + + ctx := context.Background() + require.NoError(t, redisClient.HSet(ctx, redisProjectionHealthKey(common.NodeName), map[string]interface{}{ + "degraded": 1, + "degraded_until": time.Now().Add(time.Hour).Unix(), + }).Err()) + require.NoError(t, restoreProjectionHealth()) + require.True(t, localProjectionDegraded()) + + recovered, err := recoverRedisBucketsForNode() + + require.NoError(t, err) + require.Zero(t, recovered) + require.False(t, localProjectionDegraded()) +} + +func TestLoadProjectionHealthIgnoresExpiredDegradedMarker(t *testing.T) { + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + redisServer := miniredis.RunT(t) + redisClient := redis.NewClient(&redis.Options{Addr: redisServer.Addr()}) + common.RDB = redisClient + common.RedisEnabled = true + t.Cleanup(func() { + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + _ = redisClient.Close() + }) + + ctx := context.Background() + healthKey := redisProjectionHealthKey("stale-node") + require.NoError(t, redisClient.HSet(ctx, healthKey, map[string]interface{}{ + "degraded": 1, + "degraded_until": time.Now().Add(-time.Minute).Unix(), + }).Err()) + require.NoError(t, redisClient.ZAdd(ctx, redisProjectionHealthIndex, &redis.Z{ + Score: float64(time.Now().Add(time.Hour).Unix()), + Member: healthKey, + }).Err()) + + degraded, err := loadProjectionHealth(ctx) + + require.NoError(t, err) + require.False(t, degraded) +} + +func TestQueryDetailedContextHonorsRequestCancellation(t *testing.T) { + previousDB := model.DB + previousRDB := common.RDB + previousRedisEnabled := common.RedisEnabled + db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{}) + require.NoError(t, err) + require.NoError(t, db.AutoMigrate(&model.PerfMetric{}, &model.PerfMetricFlushReceipt{})) + model.DB = db + common.RDB = nil + common.RedisEnabled = false + t.Cleanup(func() { + model.DB = previousDB + common.RDB = previousRDB + common.RedisEnabled = previousRedisEnabled + }) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, err = QueryDetailedContext(ctx, QueryParams{Model: "alpha", Hours: 1}) + + require.ErrorIs(t, err, context.Canceled) +} + +func TestBuildCoverageClampsKnownBucketEndToRequestedEnd(t *testing.T) { + coverage := buildCoverage(2_400, 2_500, map[bucketKey]counters{ + {model: "alpha", group: "prod", bucketTs: 2_480, bucketSeconds: 60}: { + requestCount: 1, + }, + }, 60) + + require.EqualValues(t, 2_500, coverage.BucketEndAt) + require.EqualValues(t, 60, coverage.BucketSeconds) + require.Equal(t, CoverageGranularityKnown, coverage.GranularityState) + require.True(t, coverage.Approximate) +} diff --git a/pkg/perf_metrics/types.go b/pkg/perf_metrics/types.go index af68eb88..13b905ee 100644 --- a/pkg/perf_metrics/types.go +++ b/pkg/perf_metrics/types.go @@ -1,6 +1,9 @@ package perfmetrics -import "sync/atomic" +import ( + "sync" + "sync/atomic" +) type Store interface { Record(sample Sample) @@ -19,9 +22,10 @@ type Sample struct { } type QueryParams struct { - Model string - Group string - Hours int + Model string + Group string + Hours int + AllowedGroups []string } type BucketPoint struct { @@ -32,6 +36,44 @@ type BucketPoint struct { AvgTps float64 `json:"avg_tps"` } +// AggregateResult is the model-level weighted aggregate. It is derived from +// the stored counters, not by averaging already-aggregated group values. +type AggregateResult struct { + AvgTtftMs int64 `json:"avg_ttft_ms"` + AvgLatencyMs int64 `json:"avg_latency_ms"` + SuccessRate float64 `json:"success_rate"` + AvgTps float64 `json:"avg_tps"` + Series []BucketPoint `json:"series"` +} + +type QueryCoverage struct { + RequestedStartAt int64 `json:"requested_start_at"` + RequestedEndAt int64 `json:"requested_end_at"` + BucketStartAt int64 `json:"bucket_start_at"` + BucketEndAt int64 `json:"bucket_end_at"` + BucketSeconds int64 `json:"bucket_seconds"` + GranularityState CoverageGranularity `json:"granularity_state"` + Approximate bool `json:"approximate"` +} + +type CoverageGranularity string + +const ( + CoverageGranularityKnown CoverageGranularity = "known" + CoverageGranularityUnknown CoverageGranularity = "unknown" + CoverageGranularityMixed CoverageGranularity = "mixed" +) + +type CollectionState string + +const ( + CollectionStateAvailable CollectionState = "available" + CollectionStatePartial CollectionState = "partial" + CollectionStateDisabled CollectionState = "collection_disabled" + CollectionStateNoSamples CollectionState = "no_samples" + CollectionStateQueryFailed CollectionState = "query_failed" +) + type GroupResult struct { Group string `json:"group"` AvgTtftMs int64 `json:"avg_ttft_ms"` @@ -42,9 +84,20 @@ type GroupResult struct { } type QueryResult struct { - ModelName string `json:"model_name"` - SeriesSchema string `json:"series_schema"` - Groups []GroupResult `json:"groups"` + ModelName string `json:"model_name"` + SeriesSchema string `json:"series_schema"` + Summary AggregateResult `json:"summary"` + Groups []GroupResult `json:"groups"` +} + +// DetailedQueryResult is used by administrator-facing Smart Operations +// endpoints. The public pricing projection remains QueryResult-compatible; +// these fields make bucket coverage and collection state explicit for +// operational decisions. +type DetailedQueryResult struct { + QueryResult + Coverage QueryCoverage `json:"coverage"` + CollectionState CollectionState `json:"collection_state"` } type ModelSummary struct { @@ -59,10 +112,18 @@ type SummaryAllResult struct { Models []ModelSummary `json:"models"` } +type DetailedSummaryAllResult struct { + Models []ModelSummary `json:"models"` + Coverage QueryCoverage `json:"coverage"` + CollectionState CollectionState `json:"collection_state"` +} + type bucketKey struct { - model string - group string - bucketTs int64 + model string + group string + node string + bucketTs int64 + bucketSeconds int64 } type counters struct { @@ -76,6 +137,11 @@ type counters struct { } type atomicBucket struct { + mu sync.Mutex + projectionWG sync.WaitGroup + closed bool + version uint64 + projected uint64 requestCount atomic.Int64 successCount atomic.Int64 totalLatencyMs atomic.Int64 @@ -85,7 +151,21 @@ type atomicBucket struct { generationMs atomic.Int64 } -func (b *atomicBucket) add(sample Sample) { +func (b *atomicBucket) add(sample Sample) bool { + _, _, added := b.addSample(sample, false) + return added +} + +func (b *atomicBucket) addForProjection(sample Sample) (counters, uint64, bool) { + return b.addSample(sample, true) +} + +func (b *atomicBucket) addSample(sample Sample, project bool) (counters, uint64, bool) { + b.mu.Lock() + defer b.mu.Unlock() + if b.closed { + return counters{}, 0, false + } b.requestCount.Add(1) if sample.Success { b.successCount.Add(1) @@ -101,9 +181,21 @@ func (b *atomicBucket) add(sample Sample) { b.outputTokens.Add(sample.OutputTokens) b.generationMs.Add(sample.GenerationMs) } + if !project { + return counters{}, 0, true + } + b.version++ + b.projectionWG.Add(1) + return b.snapshotLocked(), b.version, true } func (b *atomicBucket) snapshot() counters { + b.mu.Lock() + defer b.mu.Unlock() + return b.snapshotLocked() +} + +func (b *atomicBucket) snapshotLocked() counters { return counters{ requestCount: b.requestCount.Load(), successCount: b.successCount.Load(), @@ -115,7 +207,32 @@ func (b *atomicBucket) snapshot() counters { } } -func (b *atomicBucket) drain() counters { +func (b *atomicBucket) finishProjection(version uint64, success bool) bool { + b.mu.Lock() + if success && version > b.projected { + b.projected = version + } + incomplete := b.projected < b.version + b.mu.Unlock() + b.projectionWG.Done() + return incomplete +} + +func (b *atomicBucket) closeAndDrain() counters { + b.mu.Lock() + if b.closed { + b.mu.Unlock() + return counters{} + } + b.closed = true + b.mu.Unlock() + + // Every sample that entered this bucket has either completed its shared + // Redis projection or recorded a projection gap before the durable flush + // proceeds. This closes the late-write-after-cleanup race. + b.projectionWG.Wait() + b.mu.Lock() + defer b.mu.Unlock() return counters{ requestCount: b.requestCount.Swap(0), successCount: b.successCount.Swap(0), @@ -127,7 +244,12 @@ func (b *atomicBucket) drain() counters { } } -func (b *atomicBucket) addCounters(c counters) { +func (b *atomicBucket) addCounters(c counters) bool { + b.mu.Lock() + defer b.mu.Unlock() + if b.closed { + return false + } if c.requestCount != 0 { b.requestCount.Add(c.requestCount) } @@ -149,4 +271,5 @@ func (b *atomicBucket) addCounters(c counters) { if c.generationMs != 0 { b.generationMs.Add(c.generationMs) } + return true } diff --git a/relay/channel/ali/adaptor.go b/relay/channel/ali/adaptor.go index 82c76c7c..c950c3c3 100644 --- a/relay/channel/ali/adaptor.go +++ b/relay/channel/ali/adaptor.go @@ -106,15 +106,15 @@ func (a *Adaptor) GetRequestURL(info *relaycommon.RelayInfo) (string, error) { case constant.RelayModeResponses: fullRequestURL = fmt.Sprintf("%s/api/v2/apps/protocols/compatible-mode/v1/responses", info.ChannelBaseUrl) case constant.RelayModeImagesGenerations: - if isSyncImageModel(info.OriginModelName) { + if isSyncImageModel(info.UpstreamModelName) { fullRequestURL = fmt.Sprintf("%s/api/v1/services/aigc/multimodal-generation/generation", info.ChannelBaseUrl) } else { fullRequestURL = fmt.Sprintf("%s/api/v1/services/aigc/text2image/image-synthesis", info.ChannelBaseUrl) } case constant.RelayModeImagesEdits: - if isOldWanModel(info.OriginModelName) { + if isOldWanModel(info.UpstreamModelName) { fullRequestURL = fmt.Sprintf("%s/api/v1/services/aigc/image2image/image-synthesis", info.ChannelBaseUrl) - } else if isWanModel(info.OriginModelName) { + } else if isWanModel(info.UpstreamModelName) { fullRequestURL = fmt.Sprintf("%s/api/v1/services/aigc/image-generation/generation", info.ChannelBaseUrl) } else { fullRequestURL = fmt.Sprintf("%s/api/v1/services/aigc/multimodal-generation/generation", info.ChannelBaseUrl) @@ -139,14 +139,14 @@ func (a *Adaptor) SetupRequestHeader(c *gin.Context, req *http.Header, info *rel req.Set("X-DashScope-Plugin", c.GetString("plugin")) } if info.RelayMode == constant.RelayModeImagesGenerations { - if isSyncImageModel(info.OriginModelName) { + if isSyncImageModel(info.UpstreamModelName) { } else { req.Set("X-DashScope-Async", "enable") } } if info.RelayMode == constant.RelayModeImagesEdits { - if isWanModel(info.OriginModelName) { + if isWanModel(info.UpstreamModelName) { req.Set("X-DashScope-Async", "enable") } req.Set("Content-Type", "application/json") @@ -179,7 +179,7 @@ func (a *Adaptor) ConvertOpenAIRequest(c *gin.Context, info *relaycommon.RelayIn func (a *Adaptor) ConvertImageRequest(c *gin.Context, info *relaycommon.RelayInfo, request dto.ImageRequest) (any, error) { if info.RelayMode == constant.RelayModeImagesGenerations { - if isSyncImageModel(info.OriginModelName) { + if isSyncImageModel(info.UpstreamModelName) { a.IsSyncImageModel = true } aliRequest, err := oaiImage2AliImageRequest(info, request, a.IsSyncImageModel) @@ -188,11 +188,11 @@ func (a *Adaptor) ConvertImageRequest(c *gin.Context, info *relaycommon.RelayInf } return aliRequest, nil } else if info.RelayMode == constant.RelayModeImagesEdits { - if isOldWanModel(info.OriginModelName) { + if isOldWanModel(info.UpstreamModelName) { return oaiFormEdit2WanxImageEdit(c, info, request) } - if isSyncImageModel(info.OriginModelName) { - if isWanModel(info.OriginModelName) { + if isSyncImageModel(info.UpstreamModelName) { + if isWanModel(info.UpstreamModelName) { a.IsSyncImageModel = false } else { a.IsSyncImageModel = true diff --git a/relay/channel/ali/adaptor_test.go b/relay/channel/ali/adaptor_test.go new file mode 100644 index 00000000..24fd17e8 --- /dev/null +++ b/relay/channel/ali/adaptor_test.go @@ -0,0 +1,52 @@ +package ali + +import ( + "net/http" + "net/http/httptest" + "testing" + + "github.com/MAX-API-Next/MAX-API/dto" + relaycommon "github.com/MAX-API-Next/MAX-API/relay/common" + "github.com/MAX-API-Next/MAX-API/relay/constant" + "github.com/gin-gonic/gin" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestMappedAliImageModelsUseUpstreamProtocol(t *testing.T) { + gin.SetMode(gin.TestMode) + tests := []struct { + name string + mode int + upstreamModel string + wantPath string + wantAsync bool + }{ + {name: "sync qwen image", mode: constant.RelayModeImagesGenerations, upstreamModel: "qwen-image", wantPath: "/api/v1/services/aigc/multimodal-generation/generation"}, + {name: "wan edit", mode: constant.RelayModeImagesEdits, upstreamModel: "wan2.6-image", wantPath: "/api/v1/services/aigc/image-generation/generation", wantAsync: true}, + {name: "async image", mode: constant.RelayModeImagesGenerations, upstreamModel: "wanx-v1", wantPath: "/api/v1/services/aigc/text2image/image-synthesis", wantAsync: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + c, _ := gin.CreateTestContext(httptest.NewRecorder()) + c.Request = httptest.NewRequest(http.MethodPost, "/v1/images", nil) + info := &relaycommon.RelayInfo{ + RelayMode: tc.mode, + OriginModelName: "customer-alias", + ChannelMeta: &relaycommon.ChannelMeta{ChannelBaseUrl: "https://dashscope.example", UpstreamModelName: tc.upstreamModel}, + } + adaptor := &Adaptor{} + url, err := adaptor.GetRequestURL(info) + require.NoError(t, err) + assert.Equal(t, "https://dashscope.example"+tc.wantPath, url) + header := http.Header{} + require.NoError(t, adaptor.SetupRequestHeader(c, &header, info)) + assert.Equal(t, tc.wantAsync, header.Get("X-DashScope-Async") == "enable") + if tc.mode == constant.RelayModeImagesGenerations { + _, err = adaptor.ConvertImageRequest(c, info, dto.ImageRequest{Model: tc.upstreamModel, Prompt: "poster"}) + require.NoError(t, err) + assert.Equal(t, !tc.wantAsync, adaptor.IsSyncImageModel) + } + }) + } +} diff --git a/relay/channel/claude/relay-claude.go b/relay/channel/claude/relay-claude.go index ceb0431c..392ee77c 100644 --- a/relay/channel/claude/relay-claude.go +++ b/relay/channel/claude/relay-claude.go @@ -49,25 +49,24 @@ func RequestOpenAI2ClaudeMessage(c *gin.Context, textRequest dto.GeneralOpenAIRe claudeTools := make([]any, 0, len(textRequest.Tools)) for _, tool := range textRequest.Tools { - if params, ok := tool.Function.Parameters.(map[string]any); ok { - claudeTool := dto.Tool{ - Name: tool.Function.Name, - Description: tool.Function.Description, - } - claudeTool.InputSchema = make(map[string]interface{}) - if params["type"] != nil { - claudeTool.InputSchema["type"] = params["type"].(string) - } - claudeTool.InputSchema["properties"] = params["properties"] - claudeTool.InputSchema["required"] = params["required"] - for s, a := range params { - if s == "type" || s == "properties" || s == "required" { - continue - } - claudeTool.InputSchema[s] = a - } - claudeTools = append(claudeTools, &claudeTool) + if tool.Type != "" && tool.Type != "function" { + continue } + params, parametersAreObject := tool.Function.Parameters.(map[string]any) + if tool.Type == "" && !parametersAreObject { + continue + } + schema := make(map[string]any, len(params)+2) + for key, value := range params { + schema[key] = value + } + if schema["type"] == nil { + schema["type"] = "object" + } + if schema["properties"] == nil { + schema["properties"] = map[string]any{} + } + claudeTools = append(claudeTools, &dto.Tool{Name: tool.Function.Name, Description: tool.Function.Description, InputSchema: schema}) } // Web search tool @@ -126,7 +125,9 @@ func RequestOpenAI2ClaudeMessage(c *gin.Context, textRequest dto.GeneralOpenAIRe Model: textRequest.Model, StopSequences: nil, Temperature: textRequest.Temperature, - Tools: claudeTools, + } + if len(claudeTools) > 0 { + claudeRequest.Tools = claudeTools } if maxTokens := textRequest.GetMaxTokens(); maxTokens > 0 { claudeRequest.MaxTokens = common.GetPointer(maxTokens) @@ -142,7 +143,7 @@ func RequestOpenAI2ClaudeMessage(c *gin.Context, textRequest dto.GeneralOpenAIRe } // 处理 tool_choice 和 parallel_tool_calls - if textRequest.ToolChoice != nil || textRequest.ParallelTooCalls != nil { + if len(claudeTools) > 0 && (textRequest.ToolChoice != nil || textRequest.ParallelTooCalls != nil) { claudeToolChoice := mapToolChoice(textRequest.ToolChoice, textRequest.ParallelTooCalls) if claudeToolChoice != nil { claudeRequest.ToolChoice = claudeToolChoice diff --git a/relay/channel/claude/relay_claude_test.go b/relay/channel/claude/relay_claude_test.go index 0d55105f..814839c4 100644 --- a/relay/channel/claude/relay_claude_test.go +++ b/relay/channel/claude/relay_claude_test.go @@ -428,6 +428,73 @@ func TestRequestOpenAI2ClaudeMessage_ReturnsErrorForMalformedToolArguments(t *te require.Contains(t, err.Error(), "tool call function arguments is not valid JSON object") } +func TestRequestOpenAI2ClaudeMessageNormalizesParameterlessToolsAndOmitsEmptyTools(t *testing.T) { + parameterless, err := RequestOpenAI2ClaudeMessage(nil, dto.GeneralOpenAIRequest{ + Model: "claude-3-5-sonnet", + Tools: []dto.ToolCallRequest{{Type: "function", Function: dto.FunctionRequest{Name: "ping"}}}, + }) + require.NoError(t, err) + parameterlessTools := parameterless.GetTools() + require.Len(t, parameterlessTools, 1) + tool, ok := parameterlessTools[0].(*dto.Tool) + require.True(t, ok) + require.Equal(t, "object", tool.InputSchema["type"]) + require.Equal(t, map[string]any{}, tool.InputSchema["properties"]) + + customSchema := map[string]any{"type": 123, "properties": map[string]any{"value": map[string]any{"type": "string"}}, "additionalProperties": false} + converted, err := RequestOpenAI2ClaudeMessage(nil, dto.GeneralOpenAIRequest{ + Model: "claude-3-5-sonnet", + Tools: []dto.ToolCallRequest{{Type: "function", Function: dto.FunctionRequest{Name: "custom", Parameters: customSchema}}}, + }) + require.NoError(t, err) + convertedTools := converted.GetTools() + require.Len(t, convertedTools, 1) + customTool := convertedTools[0].(*dto.Tool) + require.Equal(t, 123, customTool.InputSchema["type"]) + require.Equal(t, false, customTool.InputSchema["additionalProperties"]) + + unsupported, err := RequestOpenAI2ClaudeMessage(nil, dto.GeneralOpenAIRequest{ + Model: "claude-3-5-sonnet", + Tools: []dto.ToolCallRequest{{ + Type: "custom", + Function: dto.FunctionRequest{ + Name: "must-not-be-converted", + Parameters: map[string]any{"type": "object"}, + }, + }}, + }) + require.NoError(t, err) + require.Empty(t, unsupported.GetTools()) + + empty, err := RequestOpenAI2ClaudeMessage(nil, dto.GeneralOpenAIRequest{Model: "claude-3-5-sonnet"}) + require.NoError(t, err) + payload, err := common.Marshal(empty) + require.NoError(t, err) + require.NotContains(t, string(payload), `"tools"`) +} + +func TestRequestOpenAI2ClaudeMessageOmitsForcedToolChoiceWhenAllToolsAreUnsupported(t *testing.T) { + converted, err := RequestOpenAI2ClaudeMessage(nil, dto.GeneralOpenAIRequest{ + Model: "claude-3-5-sonnet", + Tools: []dto.ToolCallRequest{{ + Type: "custom", + Function: dto.FunctionRequest{ + Name: "unsupported-tool", + Parameters: map[string]any{"type": "object"}, + }, + }}, + ToolChoice: "required", + }) + require.NoError(t, err) + require.Empty(t, converted.GetTools()) + require.Nil(t, converted.ToolChoice) + + payload, err := common.Marshal(converted) + require.NoError(t, err) + require.NotContains(t, string(payload), `"tools"`) + require.NotContains(t, string(payload), `"tool_choice"`) +} + func TestRequestOpenAI2ClaudeMessage_ClaudeOpus48HighUsesAdaptiveThinking(t *testing.T) { request := dto.GeneralOpenAIRequest{ Model: "claude-opus-4-8-high", diff --git a/relay/channel/codex/adaptor.go b/relay/channel/codex/adaptor.go index c7d3cbda..d926c942 100644 --- a/relay/channel/codex/adaptor.go +++ b/relay/channel/codex/adaptor.go @@ -104,6 +104,8 @@ func (a *Adaptor) ConvertOpenAIResponsesRequest(c *gin.Context, info *relaycommo // rm max_output_tokens request.MaxOutputTokens = nil request.Temperature = nil + request.FrequencyPenalty = nil + request.PresencePenalty = nil return request, nil } diff --git a/relay/channel/codex/adaptor_test.go b/relay/channel/codex/adaptor_test.go index 9955b25d..2cbffb52 100644 --- a/relay/channel/codex/adaptor_test.go +++ b/relay/channel/codex/adaptor_test.go @@ -4,6 +4,7 @@ import ( "testing" "github.com/MAX-API-Next/MAX-API/constant" + "github.com/MAX-API-Next/MAX-API/dto" relaycommon "github.com/MAX-API-Next/MAX-API/relay/common" relayconstant "github.com/MAX-API-Next/MAX-API/relay/constant" "github.com/stretchr/testify/require" @@ -24,3 +25,24 @@ func TestGetRequestURLAlphaSearch(t *testing.T) { require.NoError(t, err) require.Equal(t, "https://chatgpt.com/backend-api/codex/alpha/search", requestURL) } + +func TestConvertOpenAIResponsesRequestDropsUnsupportedPenalties(t *testing.T) { + adaptor := &Adaptor{} + info := &relaycommon.RelayInfo{ + ChannelMeta: &relaycommon.ChannelMeta{ChannelType: constant.ChannelTypeCodex}, + RelayMode: relayconstant.RelayModeResponses, + } + frequency := 0.5 + presence := 1.5 + + converted, err := adaptor.ConvertOpenAIResponsesRequest(nil, info, dto.OpenAIResponsesRequest{ + Model: "gpt-5-codex", + FrequencyPenalty: &frequency, + PresencePenalty: &presence, + }) + require.NoError(t, err) + request, ok := converted.(dto.OpenAIResponsesRequest) + require.True(t, ok) + require.Nil(t, request.FrequencyPenalty) + require.Nil(t, request.PresencePenalty) +} diff --git a/relay/channel/ollama/dto.go b/relay/channel/ollama/dto.go index 24d2d006..a0694fcd 100644 --- a/relay/channel/ollama/dto.go +++ b/relay/channel/ollama/dto.go @@ -5,12 +5,13 @@ import ( ) type OllamaChatMessage struct { - Role string `json:"role"` - Content string `json:"content,omitempty"` - Images []string `json:"images,omitempty"` - ToolCalls []OllamaToolCall `json:"tool_calls,omitempty"` - ToolName string `json:"tool_name,omitempty"` - Thinking json.RawMessage `json:"thinking,omitempty"` + Role string `json:"role"` + Content string `json:"content,omitempty"` + Images []string `json:"images,omitempty"` + ToolCalls []OllamaToolCall `json:"tool_calls,omitempty"` + ToolName string `json:"tool_name,omitempty"` + ToolCallID string `json:"tool_call_id,omitempty"` + Thinking json.RawMessage `json:"thinking,omitempty"` } type OllamaToolFunction struct { @@ -25,6 +26,7 @@ type OllamaTool struct { } type OllamaToolCall struct { + ID string `json:"id,omitempty"` Function struct { Name string `json:"name"` Arguments interface{} `json:"arguments"` diff --git a/relay/channel/ollama/relay-ollama.go b/relay/channel/ollama/relay-ollama.go index 8722b05b..d9997acc 100644 --- a/relay/channel/ollama/relay-ollama.go +++ b/relay/channel/ollama/relay-ollama.go @@ -1,7 +1,6 @@ package ollama import ( - "encoding/json" "fmt" "io" "net/http" @@ -19,24 +18,68 @@ import ( "github.com/samber/lo" ) +func toOllamaResponseFormat(responseFormat *dto.ResponseFormat) (any, error) { + if responseFormat == nil { + return nil, nil + } + switch responseFormat.Type { + case "json", "json_object": + return "json", nil + case "json_schema": + if len(responseFormat.JsonSchema) == 0 { + return nil, nil + } + var schema dto.FormatJsonSchema + if err := common.Unmarshal(responseFormat.JsonSchema, &schema); err != nil { + return nil, fmt.Errorf("invalid ollama response format: %w", err) + } + return schema.Schema, nil + default: + return nil, nil + } +} + func openAIChatToOllamaChat(c *gin.Context, r *dto.GeneralOpenAIRequest) (*OllamaChatRequest, error) { + think := r.Think + if len(think) == 0 { + effort := r.ReasoningEffort + if len(r.Reasoning) > 0 { + var reasoning dto.Reasoning + if err := common.Unmarshal(r.Reasoning, &reasoning); err != nil { + return nil, fmt.Errorf("invalid ollama reasoning: %w", err) + } + if reasoning.Effort != "" { + effort = reasoning.Effort + } + } + if effort != "" { + var value any + switch effort { + case "none": + value = false + case "low", "medium", "high", "max": + value = effort + default: + return nil, fmt.Errorf("unsupported ollama reasoning effort %q", effort) + } + var err error + think, err = common.Marshal(value) + if err != nil { + return nil, fmt.Errorf("marshal ollama think: %w", err) + } + } + } chatReq := &OllamaChatRequest{ Model: r.Model, Stream: r.Stream, Options: map[string]any{}, - Think: r.Think, + Think: think, } - if r.ResponseFormat != nil { - if r.ResponseFormat.Type == "json" { - chatReq.Format = "json" - } else if r.ResponseFormat.Type == "json_schema" { - if len(r.ResponseFormat.JsonSchema) > 0 { - var schema any - _ = json.Unmarshal(r.ResponseFormat.JsonSchema, &schema) - chatReq.Format = schema - } - } + format, err := toOllamaResponseFormat(r.ResponseFormat) + if err != nil { + return nil, err } + chatReq.Format = format // options mapping if r.Temperature != nil { @@ -89,6 +132,7 @@ func openAIChatToOllamaChat(c *gin.Context, r *dto.GeneralOpenAIRequest) (*Ollam } chatReq.Messages = make([]OllamaChatMessage, 0, len(r.Messages)) + toolNamesByCallID := make(map[string]string) for _, m := range r.Messages { var textBuilder strings.Builder var images []string @@ -117,8 +161,22 @@ func openAIChatToOllamaChat(c *gin.Context, r *dto.GeneralOpenAIRequest) (*Ollam if len(images) > 0 { cm.Images = images } - if m.Role == "tool" && m.Name != nil { - cm.ToolName = *m.Name + if m.Role == "assistant" { + if reasoning := m.GetReasoningContent(); reasoning != "" { + thinking, err := common.Marshal(reasoning) + if err != nil { + return nil, fmt.Errorf("marshal ollama thinking: %w", err) + } + cm.Thinking = thinking + } + } + if m.Role == "tool" { + cm.ToolCallID = m.ToolCallId + if m.Name != nil { + cm.ToolName = *m.Name + } else { + cm.ToolName = toolNamesByCallID[m.ToolCallId] + } } if m.ToolCalls != nil && len(m.ToolCalls) > 0 { parsed := m.ParseToolCalls() @@ -127,15 +185,18 @@ func openAIChatToOllamaChat(c *gin.Context, r *dto.GeneralOpenAIRequest) (*Ollam for _, tc := range parsed { var args interface{} if tc.Function.Arguments != "" { - _ = json.Unmarshal([]byte(tc.Function.Arguments), &args) + _ = common.Unmarshal([]byte(tc.Function.Arguments), &args) } if args == nil { args = map[string]any{} } - oc := OllamaToolCall{} + oc := OllamaToolCall{ID: tc.ID} oc.Function.Name = tc.Function.Name oc.Function.Arguments = args calls = append(calls, oc) + if tc.ID != "" { + toolNamesByCallID[tc.ID] = tc.Function.Name + } } cm.ToolCalls = calls } @@ -175,15 +236,11 @@ func openAIToGenerate(c *gin.Context, r *dto.GeneralOpenAIRequest) (*OllamaGener gen.Suffix = s } } - if r.ResponseFormat != nil { - if r.ResponseFormat.Type == "json" { - gen.Format = "json" - } else if r.ResponseFormat.Type == "json_schema" { - var schema any - _ = json.Unmarshal(r.ResponseFormat.JsonSchema, &schema) - gen.Format = schema - } + format, err := toOllamaResponseFormat(r.ResponseFormat) + if err != nil { + return nil, err } + gen.Format = format if r.Temperature != nil { gen.Options["temperature"] = r.Temperature } @@ -510,7 +567,7 @@ func FetchOllamaVersion(baseURL, apiKey string) (string, error) { Version string `json:"version"` } - if err := json.Unmarshal(body, &versionResp); err != nil { + if err := common.Unmarshal(body, &versionResp); err != nil { return "", fmt.Errorf("解析响应失败: %v", err) } diff --git a/relay/channel/ollama/stream.go b/relay/channel/ollama/stream.go index 7c1ff379..0bcc8807 100644 --- a/relay/channel/ollama/stream.go +++ b/relay/channel/ollama/stream.go @@ -54,8 +54,12 @@ func ollamaToolCallsToOpenAI(toolCalls []OllamaToolCall, startIndex int, include argBytes = data } } + toolCallID := tc.ID + if toolCallID == "" { + toolCallID = fmt.Sprintf("call_%d", startIndex) + } toolCall := dto.ToolCallResponse{ - ID: fmt.Sprintf("call_%d", startIndex), + ID: toolCallID, Type: "function", Function: dto.FunctionResponse{ Name: tc.Function.Name, diff --git a/relay/channel/ollama/stream_test.go b/relay/channel/ollama/stream_test.go index 42cfc4b6..977055f3 100644 --- a/relay/channel/ollama/stream_test.go +++ b/relay/channel/ollama/stream_test.go @@ -131,6 +131,42 @@ func TestOllamaRequestConvertersOmitUnsetStream(t *testing.T) { require.JSONEq(t, `{"model":"llama-test"}`, string(generatePayload)) } +func TestOllamaChatConverterPreservesReasoningAndToolContext(t *testing.T) { + gin.SetMode(gin.TestMode) + c, _ := gin.CreateTestContext(httptest.NewRecorder()) + reasoning := "internal plan" + rawCalls, err := common.Marshal([]dto.ToolCallRequest{{ + ID: "call_weather", Type: "function", Function: dto.FunctionRequest{Name: "weather", Arguments: `{"city":"Paris"}`}, + }}) + require.NoError(t, err) + request := &dto.GeneralOpenAIRequest{ + Model: "llama-test", + ReasoningEffort: "high", + Messages: []dto.Message{ + {Role: "assistant", ReasoningContent: &reasoning, ToolCalls: rawCalls}, + {Role: "tool", ToolCallId: "call_weather", Content: "sunny"}, + }, + } + converted, err := openAIChatToOllamaChat(c, request) + require.NoError(t, err) + require.JSONEq(t, `"high"`, string(converted.Think)) + require.Len(t, converted.Messages, 2) + require.JSONEq(t, `"internal plan"`, string(converted.Messages[0].Thinking)) + require.Equal(t, "call_weather", converted.Messages[0].ToolCalls[0].ID) + require.Equal(t, "call_weather", converted.Messages[1].ToolCallID) + require.Equal(t, "weather", converted.Messages[1].ToolName) +} + +func TestOllamaConvertersRejectMalformedJSONSchema(t *testing.T) { + gin.SetMode(gin.TestMode) + c, _ := gin.CreateTestContext(httptest.NewRecorder()) + request := &dto.GeneralOpenAIRequest{Model: "llama-test", ResponseFormat: &dto.ResponseFormat{Type: "json_schema", JsonSchema: []byte(`{`)}} + _, err := openAIChatToOllamaChat(c, request) + require.ErrorContains(t, err, "invalid ollama response format") + _, err = openAIToGenerate(c, request) + require.ErrorContains(t, err, "invalid ollama response format") +} + func TestOllamaStreamHandlerToolCalls(t *testing.T) { gin.SetMode(gin.TestMode) diff --git a/relay/common/override.go b/relay/common/override.go index ae24493c..8d116d09 100644 --- a/relay/common/override.go +++ b/relay/common/override.go @@ -30,6 +30,11 @@ var paramOverrideSensitivePathPrefixes = []string{ "model", "original_model", "upstream_model", + "reasoning", + "reasoning_effort", + "output_config", + "generationConfig.thinkingConfig", + "generation_config.thinking_config", "service_tier", "inference_geo", "speed", @@ -191,6 +196,7 @@ func ApplyParamOverrideWithRelayInfo(jsonData []byte, info *RelayInfo) ([]byte, if err != nil { return nil, err } + syncReasoningEffortAfterParamOverride(info, jsonData, result) syncRuntimeHeaderOverrideFromContext(info, overrideCtx) if info != nil { if recorder != nil { @@ -202,6 +208,48 @@ func ApplyParamOverrideWithRelayInfo(jsonData []byte, info *RelayInfo) ([]byte, return result, nil } +func syncReasoningEffortAfterParamOverride(info *RelayInfo, before, after []byte) { + if info == nil { + return + } + _, existedBefore := extractReasoningEffortFromJSON(info.GetFinalRequestRelayFormat(), before) + effort, existsAfter := extractReasoningEffortFromJSON(info.GetFinalRequestRelayFormat(), after) + if existsAfter { + info.SetReasoningEffort(effort) + return + } + if existedBefore { + info.SetReasoningEffort("") + } +} + +func extractReasoningEffortFromJSON(format types.RelayFormat, data []byte) (string, bool) { + var paths []string + switch format { + case types.RelayFormatOpenAI: + paths = []string{"reasoning_effort", "reasoning.effort"} + case types.RelayFormatOpenAIResponses: + paths = []string{"reasoning.effort"} + case types.RelayFormatClaude: + paths = []string{"output_config.effort"} + case types.RelayFormatGemini: + paths = []string{"generationConfig.thinkingConfig.thinkingLevel", "generation_config.thinking_config.thinking_level"} + default: + return "", false + } + for _, path := range paths { + value := gjson.GetBytes(data, path) + if !value.Exists() { + continue + } + if value.Type != gjson.String { + return "", true + } + return strings.TrimSpace(value.String()), true + } + return "", false +} + func shouldEnableParamOverrideAudit(paramOverride map[string]interface{}) bool { if common.DebugEnabled { return true diff --git a/relay/common/override_test.go b/relay/common/override_test.go index c42121b8..b8f17a24 100644 --- a/relay/common/override_test.go +++ b/relay/common/override_test.go @@ -2210,6 +2210,29 @@ func TestApplyParamOverrideWithRelayInfoRecordsOnlyKeyOperationsWhenDebugDisable } } +func TestApplyParamOverrideWithRelayInfoSynchronizesReasoningEffort(t *testing.T) { + tests := []struct { + name string + format types.RelayFormat + before string + override map[string]interface{} + wantEffort string + }{ + {name: "chat set none", format: types.RelayFormatOpenAI, before: `{"model":"gpt"}`, override: map[string]interface{}{"reasoning_effort": "none"}, wantEffort: "none"}, + {name: "chat change", format: types.RelayFormatOpenAI, before: `{"model":"gpt","reasoning_effort":"low"}`, override: map[string]interface{}{"reasoning_effort": "high"}, wantEffort: "high"}, + {name: "chat delete", format: types.RelayFormatOpenAI, before: `{"model":"gpt","reasoning_effort":"low"}`, override: map[string]interface{}{"operations": []interface{}{map[string]interface{}{"path": "reasoning_effort", "mode": "delete"}}}, wantEffort: ""}, + {name: "responses change", format: types.RelayFormatOpenAIResponses, before: `{"model":"gpt","reasoning":{"effort":"medium"}}`, override: map[string]interface{}{"operations": []interface{}{map[string]interface{}{"path": "reasoning.effort", "mode": "set", "value": "max"}}}, wantEffort: "max"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + info := &RelayInfo{RelayFormat: tc.format, ReasoningEffort: "stale", ChannelMeta: &ChannelMeta{ParamOverride: tc.override}} + _, err := ApplyParamOverrideWithRelayInfo([]byte(tc.before), info) + require.NoError(t, err) + require.Equal(t, tc.wantEffort, info.ReasoningEffort) + }) + } +} + func TestApplyParamOverrideWithRelayInfoRecordsConversationBodyOperationsWhenDebugDisabled(t *testing.T) { originalDebugEnabled := common2.DebugEnabled common2.DebugEnabled = false diff --git a/relay/common/relay_info.go b/relay/common/relay_info.go index 286f41af..57c07c7e 100644 --- a/relay/common/relay_info.go +++ b/relay/common/relay_info.go @@ -240,6 +240,11 @@ func (info *RelayInfo) InitChannelMeta(c *gin.Context) { } info.ChannelMeta = channelMeta + if model_setting.GetGlobalSettings().PassThroughRequestEnabled || channelMeta.ChannelSetting.PassThroughBodyEnabled { + info.ReasoningEffort = "" + } else { + info.ReasoningEffort = reasoningEffortFromRequest(info.Request) + } // reset some fields based on channel meta // 重置某些字段,例如模型名称等 @@ -438,6 +443,36 @@ func GenRelayInfoOpenAI(c *gin.Context, request dto.Request) *RelayInfo { return info } +func reasoningEffortFromRequest(request dto.Request) string { + var effort string + switch req := request.(type) { + case *dto.GeneralOpenAIRequest: + if req == nil { + return "" + } + effort = req.ReasoningEffort + if strings.TrimSpace(effort) == "" && len(req.Reasoning) > 0 { + value := gjson.GetBytes(req.Reasoning, "effort") + if value.Type == gjson.String { + effort = value.String() + } + } + case *dto.OpenAIResponsesRequest: + if req != nil && req.Reasoning != nil { + effort = req.Reasoning.Effort + } + case *dto.ClaudeRequest: + if req != nil { + effort = req.GetEfforts() + } + case *dto.GeminiChatRequest: + if req != nil && req.GenerationConfig.ThinkingConfig != nil { + effort = req.GenerationConfig.ThinkingConfig.ThinkingLevel + } + } + return strings.TrimSpace(effort) +} + func genBaseRelayInfo(c *gin.Context, request dto.Request) *RelayInfo { //channelType := common.GetContextKeyInt(c, constant.ContextKeyChannelType) @@ -469,7 +504,8 @@ func genBaseRelayInfo(c *gin.Context, request dto.Request) *RelayInfo { reqId = common.GetTimeString() + common.GetRandomString(8) } info := &RelayInfo{ - Request: request, + Request: request, + ReasoningEffort: reasoningEffortFromRequest(request), RequestId: reqId, UserId: common.GetContextKeyInt(c, constant.ContextKeyUserId), @@ -647,6 +683,13 @@ func (info *RelayInfo) GetFinalRequestRelayFormat() types.RelayFormat { return info.RelayFormat } +func (info *RelayInfo) SetReasoningEffort(effort string) { + if info == nil { + return + } + info.ReasoningEffort = strings.TrimSpace(effort) +} + func GenRelayInfoResponsesCompaction(c *gin.Context, request *dto.OpenAIResponsesCompactionRequest) *RelayInfo { info := genBaseRelayInfo(c, request) if info.RelayMode == relayconstant.RelayModeUnknown { diff --git a/relay/common/relay_info_test.go b/relay/common/relay_info_test.go index b80c1525..3cf83d0b 100644 --- a/relay/common/relay_info_test.go +++ b/relay/common/relay_info_test.go @@ -4,10 +4,30 @@ import ( "testing" "github.com/MAX-API-Next/MAX-API/common" + "github.com/MAX-API-Next/MAX-API/dto" "github.com/MAX-API-Next/MAX-API/types" "github.com/stretchr/testify/require" ) +func TestReasoningEffortFromRequest(t *testing.T) { + chatReasoning, err := common.Marshal(map[string]any{"effort": " high "}) + require.NoError(t, err) + tests := []struct { + name string + request dto.Request + want string + }{ + {name: "chat scalar", request: &dto.GeneralOpenAIRequest{ReasoningEffort: " low "}, want: "low"}, + {name: "chat object", request: &dto.GeneralOpenAIRequest{Reasoning: chatReasoning}, want: "high"}, + {name: "responses", request: &dto.OpenAIResponsesRequest{Reasoning: &dto.Reasoning{Effort: "medium"}}, want: "medium"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + require.Equal(t, tc.want, reasoningEffortFromRequest(tc.request)) + }) + } +} + func TestRelayInfoGetFinalRequestRelayFormatPrefersExplicitFinal(t *testing.T) { info := &RelayInfo{ RelayFormat: types.RelayFormatOpenAI, diff --git a/router/api-router.go b/router/api-router.go index 1b1a4f0f..a2452325 100644 --- a/router/api-router.go +++ b/router/api-router.go @@ -227,6 +227,15 @@ func SetApiRouter(router *gin.Engine) { performanceRoute.GET("/logs", controller.GetLogFiles) performanceRoute.DELETE("/logs", controller.CleanupLogFiles) } + smartOpsRoute := apiRouter.Group("/smart-ops") + smartOpsRoute.Use(middleware.AdminAuth()) + { + smartOpsRoute.GET("/alerts", controller.GetSmartOpsAlerts) + smartOpsRoute.GET("/channel-performance", controller.GetChannelPerformance) + smartOpsRoute.GET("/channel-performance/detail", controller.GetChannelPerformanceDetail) + smartOpsRoute.GET("/model-performance", controller.GetModelPerformance) + smartOpsRoute.GET("/model-performance/detail", controller.GetModelPerformanceDetail) + } ratioSyncRoute := apiRouter.Group("/ratio_sync") ratioSyncRoute.Use(middleware.RootAuth()) { diff --git a/router/relay-router.go b/router/relay-router.go index 091c8d2f..739e3849 100644 --- a/router/relay-router.go +++ b/router/relay-router.go @@ -25,7 +25,7 @@ func SetRelayRouter(router *gin.Engine) { case c.GetHeader("x-api-key") != "" && c.GetHeader("anthropic-version") != "": controller.ListModels(c, constant.ChannelTypeAnthropic) case c.GetHeader("x-goog-api-key") != "" || c.Query("key") != "": // 单独的适配 - controller.RetrieveModel(c, constant.ChannelTypeGemini) + controller.ListModels(c, constant.ChannelTypeGemini) default: controller.ListModels(c, constant.ChannelTypeOpenAI) } diff --git a/router/relay_router_test.go b/router/relay_router_test.go index 86087201..1ed2c752 100644 --- a/router/relay_router_test.go +++ b/router/relay_router_test.go @@ -1,11 +1,19 @@ package router import ( + "fmt" "net/http" + "net/http/httptest" + "strings" "testing" + "github.com/MAX-API-Next/MAX-API/common" + "github.com/MAX-API-Next/MAX-API/model" "github.com/gin-gonic/gin" + "github.com/glebarez/sqlite" + "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "gorm.io/gorm" ) func TestRelayRouterRegistersAlphaSearch(t *testing.T) { @@ -23,3 +31,76 @@ func TestRelayRouterRegistersAlphaSearch(t *testing.T) { require.True(t, found, "expected POST /v1/alpha/search route to be registered") } + +func TestListModelsSupportsGeminiAuthenticationOnV1Route(t *testing.T) { + setupRelayModelsTestDB(t) + user := model.User{Username: "models-user", Status: common.UserStatusEnabled, Group: "default", Quota: 100} + require.NoError(t, model.DB.Create(&user).Error) + require.NoError(t, model.DB.Create(&model.Token{ + UserId: user.Id, + Key: "modelstestkey", + Status: common.TokenStatusEnabled, + ExpiredTime: -1, + UnlimitedQuota: true, + ModelLimitsEnabled: true, + ModelLimits: "[]", + }).Error) + + engine := gin.New() + SetRelayRouter(engine) + for _, tc := range []struct { + name string + path string + header string + }{ + {name: "query key", path: "/v1/models?key=modelstestkey"}, + {name: "Gemini header", path: "/v1/models", header: "modelstestkey"}, + } { + t.Run(tc.name, func(t *testing.T) { + recorder := httptest.NewRecorder() + request := httptest.NewRequest(http.MethodGet, tc.path, nil) + if tc.header != "" { + request.Header.Set("x-goog-api-key", tc.header) + } + engine.ServeHTTP(recorder, request) + require.Equal(t, http.StatusOK, recorder.Code) + var payload map[string]any + require.NoError(t, common.Unmarshal(recorder.Body.Bytes(), &payload)) + assert.Contains(t, payload, "models") + assert.NotContains(t, payload, "error") + }) + } +} + +func setupRelayModelsTestDB(t *testing.T) { + t.Helper() + gin.SetMode(gin.TestMode) + originalDB := model.DB + originalLogDB := model.LOG_DB + originalRedisEnabled := common.RedisEnabled + originalUsingSQLite := common.UsingSQLite + originalUsingMySQL := common.UsingMySQL + originalUsingPostgreSQL := common.UsingPostgreSQL + + common.RedisEnabled = false + common.UsingSQLite = true + common.UsingMySQL = false + common.UsingPostgreSQL = false + dsn := fmt.Sprintf("file:%s?mode=memory&cache=shared", strings.ReplaceAll(t.Name(), "/", "_")) + db, err := gorm.Open(sqlite.Open(dsn), &gorm.Config{}) + require.NoError(t, err) + model.DB = db + model.LOG_DB = db + require.NoError(t, db.AutoMigrate(&model.User{}, &model.Token{})) + t.Cleanup(func() { + if sqlDB, err := db.DB(); err == nil { + _ = sqlDB.Close() + } + model.DB = originalDB + model.LOG_DB = originalLogDB + common.RedisEnabled = originalRedisEnabled + common.UsingSQLite = originalUsingSQLite + common.UsingMySQL = originalUsingMySQL + common.UsingPostgreSQL = originalUsingPostgreSQL + }) +} diff --git a/service/channel_performance.go b/service/channel_performance.go new file mode 100644 index 00000000..586ad03d --- /dev/null +++ b/service/channel_performance.go @@ -0,0 +1,291 @@ +package service + +import ( + "context" + "errors" + "fmt" + "time" + + "github.com/MAX-API-Next/MAX-API/common" + "github.com/MAX-API-Next/MAX-API/constant" + "github.com/MAX-API-Next/MAX-API/model" +) + +var ErrInvalidChannelPerformanceQuery = errors.New("invalid channel performance query") + +const ( + channelPerformanceDetailHours = 24 +) + +type ChannelPerformanceQuery = model.ChannelPerformanceQuery + +type ChannelPerformanceTimeRange struct { + StartAt int64 `json:"start_at"` + EndAt int64 `json:"end_at"` + Hours int `json:"hours"` +} + +type ChannelPerformanceSummaryStats struct { + ChannelCount int `json:"channel_count"` + ObservedCount int64 `json:"observed_count"` + ConsumeLogCount int64 `json:"consume_log_count"` + ErrorLogCount int64 `json:"error_log_count"` + ConsumedQuota int64 `json:"consumed_quota"` + RetryLogCount *int64 `json:"retry_log_count"` + LatencySampleCount int64 `json:"latency_sample_count"` + ObservedSuccessRate *float64 `json:"observed_success_rate"` + AvgLoggedLatencyMs *float64 `json:"avg_logged_latency_ms"` + LastObservedAt int64 `json:"last_observed_at"` +} + +type ChannelPerformanceItem struct { + ChannelID int `json:"channel_id"` + ChannelName string `json:"channel_name"` + ChannelType *int `json:"channel_type"` + ChannelStatus *int `json:"channel_status"` + ModelName string `json:"model_name"` + EffectiveGroup string `json:"effective_group"` + ObservedCount int64 `json:"observed_count"` + ConsumeLogCount int64 `json:"consume_log_count"` + ErrorLogCount int64 `json:"error_log_count"` + ConsumedQuota int64 `json:"consumed_quota"` + RetryLogCount *int64 `json:"retry_log_count"` + LatencySampleCount int64 `json:"latency_sample_count"` + ObservedSuccessRate *float64 `json:"observed_success_rate"` + AvgLoggedLatencyMs *float64 `json:"avg_logged_latency_ms"` + LastObservedAt int64 `json:"last_observed_at"` + ProbeLatencyMs *int `json:"probe_latency_ms"` + ProbeTestTime *int64 `json:"probe_test_time"` + QualityFlags []string `json:"quality_flags"` +} + +type ChannelPerformanceSummary struct { + StorageMode string `json:"storage_mode"` + Source string `json:"source"` + Partial bool `json:"partial"` + QualityFlags []string `json:"quality_flags"` + TimeRange ChannelPerformanceTimeRange `json:"time_range"` + Summary ChannelPerformanceSummaryStats `json:"summary"` + Items []ChannelPerformanceItem `json:"items"` + Truncated bool `json:"truncated"` + GeneratedAt int64 `json:"generated_at"` +} + +// ChannelPerformanceReader is deliberately read-only and narrow. Future +// Attempt/Evidence implementations can replace the legacy source without +// changing the controller or the Smart Operations Center UI contract. +type ChannelPerformanceReader interface { + Summary(context.Context, ChannelPerformanceQuery) (ChannelPerformanceSummary, error) +} + +type legacyChannelPerformanceReader struct{} + +func (legacyChannelPerformanceReader) Summary(ctx context.Context, query ChannelPerformanceQuery) (ChannelPerformanceSummary, error) { + query, rangeWasClamped, err := normalizeChannelPerformanceQuery(query) + if err != nil { + return ChannelPerformanceSummary{}, err + } + + legacyResult, err := model.QueryLegacyChannelPerformance(ctx, query) + if err != nil { + return ChannelPerformanceSummary{}, err + } + rows := legacyResult.Rows + limit := query.Limit + truncated := len(rows) > limit + if truncated { + rows = rows[:limit] + } + + ids := make([]int, 0, len(rows)) + seen := make(map[int]struct{}, len(rows)) + for _, row := range rows { + if _, ok := seen[row.ChannelID]; ok { + continue + } + seen[row.ChannelID] = struct{}{} + ids = append(ids, row.ChannelID) + } + channels, err := model.GetChannelPerformanceChannels(ctx, ids) + if err != nil { + return ChannelPerformanceSummary{}, err + } + return buildLegacyChannelPerformanceSummary( + query, + rangeWasClamped, + legacyResult, + rows, + channels, + truncated, + common.LogConsumeEnabled, + constant.ErrorLogEnabled, + time.Now().Unix(), + ), nil +} + +func buildLegacyChannelPerformanceSummary( + query ChannelPerformanceQuery, + rangeWasClamped bool, + legacyResult model.LegacyChannelPerformanceResult, + rows []model.LegacyChannelPerformanceRow, + channels map[int]model.ChannelPerformanceChannel, + truncated bool, + consumeLogsEnabled bool, + errorLogsEnabled bool, + generatedAt int64, +) ChannelPerformanceSummary { + qualityFlags := []string{ + "legacy", + "partial", + "weak_correlation", + "heuristic_outcome", + "coarse_latency", + "probe_separate", + "non_attempt_logs_excluded", + "incomplete_errors_possible", + } + if rangeWasClamped { + qualityFlags = append(qualityFlags, "time_range_clamped") + } + if truncated { + qualityFlags = append(qualityFlags, "truncated") + } + switch legacyResult.RetryMetricState { + case model.LegacyRetryMetricStatePending: + qualityFlags = append(qualityFlags, "retry_backfill_pending") + case model.LegacyRetryMetricStateUnavailable: + qualityFlags = append(qualityFlags, "retry_metrics_unavailable") + } + if !consumeLogsEnabled { + qualityFlags = append(qualityFlags, "consume_logs_disabled") + } + if !errorLogsEnabled { + qualityFlags = append(qualityFlags, "error_logs_disabled") + } + + allowSuccessRate := consumeLogsEnabled && errorLogsEnabled + items := make([]ChannelPerformanceItem, 0, len(rows)) + for _, row := range rows { + metadata, metadataFound := channels[row.ChannelID] + itemFlags := append([]string(nil), qualityFlags...) + item := ChannelPerformanceItem{ + ChannelID: row.ChannelID, + ChannelName: fmt.Sprintf("Channel #%d", row.ChannelID), + ModelName: row.ModelName, + EffectiveGroup: row.EffectiveGroup, + ObservedCount: row.ObservedCount, + ConsumeLogCount: row.ConsumeLogCount, + ErrorLogCount: row.ErrorLogCount, + ConsumedQuota: row.ConsumedQuota, + LatencySampleCount: row.LatencySampleCount, + LastObservedAt: row.LastObservedAt, + QualityFlags: itemFlags, + } + if legacyResult.RetryMetricsAvailable { + item.RetryLogCount = int64Pointer(row.RetryLogCount) + } + if metadataFound { + item.ChannelName = metadata.Name + item.ChannelType = intPointer(metadata.Type) + item.ChannelStatus = intPointer(metadata.Status) + if metadata.ResponseTime > 0 { + item.ProbeLatencyMs = intPointer(metadata.ResponseTime) + } + if metadata.TestTime > 0 { + item.ProbeTestTime = int64Pointer(metadata.TestTime) + } + } else { + item.QualityFlags = append(item.QualityFlags, "metadata_missing") + } + if item.ProbeLatencyMs == nil { + item.QualityFlags = append(item.QualityFlags, "probe_unavailable") + } + if item.LatencySampleCount > 0 { + item.AvgLoggedLatencyMs = floatPointer(float64(row.TotalUseTimeSeconds*1000) / float64(item.LatencySampleCount)) + } + if allowSuccessRate && item.ObservedCount > 0 { + item.ObservedSuccessRate = floatPointer(float64(item.ConsumeLogCount) / float64(item.ObservedCount) * 100) + } + items = append(items, item) + } + + summary := ChannelPerformanceSummaryStats{ + ChannelCount: int(legacyResult.Summary.ChannelCount), + ObservedCount: legacyResult.Summary.ObservedCount, + ConsumeLogCount: legacyResult.Summary.ConsumeLogCount, + ErrorLogCount: legacyResult.Summary.ErrorLogCount, + ConsumedQuota: legacyResult.Summary.ConsumedQuota, + LatencySampleCount: legacyResult.Summary.LatencySampleCount, + LastObservedAt: legacyResult.Summary.LastObservedAt, + } + if legacyResult.RetryMetricsAvailable { + summary.RetryLogCount = int64Pointer(legacyResult.Summary.RetryLogCount) + } + if allowSuccessRate && summary.ObservedCount > 0 { + summary.ObservedSuccessRate = floatPointer(float64(summary.ConsumeLogCount) / float64(summary.ObservedCount) * 100) + } + if summary.LatencySampleCount > 0 { + summary.AvgLoggedLatencyMs = floatPointer(float64(legacyResult.Summary.TotalUseTimeSeconds*1000) / float64(summary.LatencySampleCount)) + } + + return ChannelPerformanceSummary{ + StorageMode: "legacy_log", + Source: "log_db", + Partial: true, + QualityFlags: qualityFlags, + TimeRange: ChannelPerformanceTimeRange{ + StartAt: query.StartAt, + EndAt: query.EndAt, + Hours: int((query.EndAt - query.StartAt) / int64(time.Hour/time.Second)), + }, + Summary: summary, + Items: items, + Truncated: truncated, + GeneratedAt: generatedAt, + } +} + +var defaultChannelPerformanceReader ChannelPerformanceReader = legacyChannelPerformanceReader{} + +func GetChannelPerformance(ctx context.Context, query ChannelPerformanceQuery) (ChannelPerformanceSummary, error) { + return defaultChannelPerformanceReader.Summary(ctx, query) +} + +// GetChannelPerformanceDetail returns a channel-scoped, read-only 24-hour +// log projection. Keeping this contract separate from the list filters avoids +// accidental automatic refreshes and lets a future Attempt-backed reader +// replace the legacy source without changing the administrator API. +func GetChannelPerformanceDetail(ctx context.Context, channelID int) (ChannelPerformanceSummary, error) { + if channelID <= 0 { + return ChannelPerformanceSummary{}, fmt.Errorf("%w: channel_id must be greater than zero", ErrInvalidChannelPerformanceQuery) + } + return defaultChannelPerformanceReader.Summary(ctx, ChannelPerformanceQuery{ + ChannelID: channelID, + Hours: channelPerformanceDetailHours, + Limit: maxPerformanceLimit, + }) +} + +func normalizeChannelPerformanceQuery(query ChannelPerformanceQuery) (ChannelPerformanceQuery, bool, error) { + normalized, rangeWasClamped, err := normalizePerformanceQuery( + query.StartAt, + query.EndAt, + query.Hours, + query.Limit, + ErrInvalidChannelPerformanceQuery, + ) + if err != nil { + return query, false, err + } + query.StartAt = normalized.StartAt + query.EndAt = normalized.EndAt + query.Hours = normalized.Hours + query.Limit = normalized.Limit + return query, rangeWasClamped, nil +} + +func intPointer(value int) *int { return &value } + +func int64Pointer(value int64) *int64 { return &value } + +func floatPointer(value float64) *float64 { return &value } diff --git a/service/channel_performance_test.go b/service/channel_performance_test.go new file mode 100644 index 00000000..dae2034f --- /dev/null +++ b/service/channel_performance_test.go @@ -0,0 +1,185 @@ +package service + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/MAX-API-Next/MAX-API/model" + "github.com/stretchr/testify/require" +) + +type capturingChannelPerformanceReader struct { + query ChannelPerformanceQuery +} + +func (reader *capturingChannelPerformanceReader) Summary(_ context.Context, query ChannelPerformanceQuery) (ChannelPerformanceSummary, error) { + reader.query = query + return ChannelPerformanceSummary{TimeRange: ChannelPerformanceTimeRange{Hours: query.Hours}}, nil +} + +func TestNormalizeChannelPerformanceQueryBoundsWindowAndLimit(t *testing.T) { + end := int64(1_700_000_000) + query, clamped, err := normalizeChannelPerformanceQuery(ChannelPerformanceQuery{ + StartAt: end - int64(30*24*time.Hour/time.Second), + EndAt: end, + Limit: 999, + }) + + require.NoError(t, err) + require.True(t, clamped) + require.Equal(t, maxPerformanceLimit, query.Limit) + require.EqualValues(t, maxPerformanceHours, (query.EndAt-query.StartAt)/int64(time.Hour/time.Second)) +} + +func TestNormalizeChannelPerformanceQueryUsesServerOwnedHourWindow(t *testing.T) { + before := time.Now().Unix() + query, clamped, err := normalizeChannelPerformanceQuery(ChannelPerformanceQuery{}) + after := time.Now().Unix() + + require.NoError(t, err) + require.False(t, clamped) + require.GreaterOrEqual(t, query.EndAt, before) + require.LessOrEqual(t, query.EndAt, after) + require.EqualValues(t, time.Hour/time.Second, query.EndAt-query.StartAt) + + query, clamped, err = normalizeChannelPerformanceQuery(ChannelPerformanceQuery{Hours: 37}) + require.NoError(t, err) + require.False(t, clamped) + require.EqualValues(t, 37*time.Hour/time.Second, query.EndAt-query.StartAt) + + query, clamped, err = normalizeChannelPerformanceQuery(ChannelPerformanceQuery{Hours: 999}) + require.NoError(t, err) + require.True(t, clamped) + require.EqualValues(t, maxPerformanceHours*time.Hour/time.Second, query.EndAt-query.StartAt) +} + +func TestNormalizeChannelPerformanceQueryRejectsReversedRange(t *testing.T) { + _, _, err := normalizeChannelPerformanceQuery(ChannelPerformanceQuery{ + StartAt: 20, + EndAt: 10, + }) + + require.Error(t, err) + require.True(t, errors.Is(err, ErrInvalidChannelPerformanceQuery)) +} + +func TestGetChannelPerformanceDetailUsesFixedChannelWindow(t *testing.T) { + previousReader := defaultChannelPerformanceReader + reader := &capturingChannelPerformanceReader{} + defaultChannelPerformanceReader = reader + t.Cleanup(func() { defaultChannelPerformanceReader = previousReader }) + + result, err := GetChannelPerformanceDetail(context.Background(), 17) + + require.NoError(t, err) + require.Equal(t, 17, reader.query.ChannelID) + require.Equal(t, channelPerformanceDetailHours, reader.query.Hours) + require.Equal(t, maxPerformanceLimit, reader.query.Limit) + require.Equal(t, channelPerformanceDetailHours, result.TimeRange.Hours) +} + +func TestGetChannelPerformanceDetailRejectsInvalidChannel(t *testing.T) { + _, err := GetChannelPerformanceDetail(context.Background(), 0) + + require.Error(t, err) + require.ErrorIs(t, err, ErrInvalidChannelPerformanceQuery) +} + +func TestBuildLegacyChannelPerformanceSummaryPreservesFullSummaryContract(t *testing.T) { + retryCount := int64(2) + result := buildLegacyChannelPerformanceSummary( + ChannelPerformanceQuery{StartAt: 1_700_000_000, EndAt: 1_700_003_600, Limit: 1}, + false, + model.LegacyChannelPerformanceResult{ + Rows: []model.LegacyChannelPerformanceRow{ + {ChannelID: 7, ModelName: "alpha", EffectiveGroup: "prod"}, + {ChannelID: 8, ModelName: "beta", EffectiveGroup: "prod"}, + }, + Summary: model.LegacyChannelPerformanceSummary{ + ChannelCount: 2, + ObservedCount: 5, + ConsumeLogCount: 3, + ErrorLogCount: 2, + ConsumedQuota: 900, + RetryLogCount: retryCount, + LatencySampleCount: 4, + TotalUseTimeSeconds: 3, + LastObservedAt: 1_700_003_500, + }, + RetryMetricsAvailable: true, + RetryMetricState: model.LegacyRetryMetricStateAvailable, + }, + []model.LegacyChannelPerformanceRow{ + { + ChannelID: 7, + ModelName: "alpha", + EffectiveGroup: "prod", + ObservedCount: 3, + ConsumeLogCount: 2, + ErrorLogCount: 1, + ConsumedQuota: 500, + RetryLogCount: 1, + LatencySampleCount: 2, + TotalUseTimeSeconds: 1, + LastObservedAt: 1_700_003_400, + }, + }, + map[int]model.ChannelPerformanceChannel{ + 7: {ID: 7, Name: "Primary", Type: 1, Status: 1}, + }, + true, + true, + true, + 1_700_003_600, + ) + + require.True(t, result.Truncated) + require.Len(t, result.Items, 1) + require.Contains(t, result.QualityFlags, "truncated") + require.Equal(t, 2, result.Summary.ChannelCount, "summary must not be reduced to the limited row set") + require.EqualValues(t, 5, result.Summary.ObservedCount) + require.EqualValues(t, 3, result.Summary.ConsumeLogCount) + require.EqualValues(t, 900, result.Summary.ConsumedQuota) + require.NotNil(t, result.Summary.RetryLogCount) + require.EqualValues(t, retryCount, *result.Summary.RetryLogCount) + require.NotNil(t, result.Summary.ObservedSuccessRate) + require.InDelta(t, 60, *result.Summary.ObservedSuccessRate, 0.001) + require.NotNil(t, result.Summary.AvgLoggedLatencyMs) + require.InDelta(t, 750, *result.Summary.AvgLoggedLatencyMs, 0.001, "latency must use the explicit sample count") + require.EqualValues(t, 1_700_003_500, result.Summary.LastObservedAt) + require.EqualValues(t, 1_700_003_600, result.GeneratedAt) + + item := result.Items[0] + require.Equal(t, "Primary", item.ChannelName) + require.NotNil(t, item.ObservedSuccessRate) + require.EqualValues(t, 500, item.ConsumedQuota) + require.InDelta(t, 66.666, *item.ObservedSuccessRate, 0.01) + require.NotNil(t, item.AvgLoggedLatencyMs) + require.InDelta(t, 500, *item.AvgLoggedLatencyMs, 0.001) +} + +func TestBuildLegacyChannelPerformanceSummaryDoesNotInventRetryMetrics(t *testing.T) { + result := buildLegacyChannelPerformanceSummary( + ChannelPerformanceQuery{StartAt: 1, EndAt: 3_601, Limit: 10}, + false, + model.LegacyChannelPerformanceResult{ + Summary: model.LegacyChannelPerformanceSummary{ObservedCount: 1, ConsumeLogCount: 1}, + RetryMetricState: model.LegacyRetryMetricStatePending, + }, + []model.LegacyChannelPerformanceRow{{ + ChannelID: 1, ModelName: "alpha", ObservedCount: 1, ConsumeLogCount: 1, + }}, + nil, + false, + true, + true, + 3_601, + ) + + require.Nil(t, result.Summary.RetryLogCount) + require.Nil(t, result.Items[0].RetryLogCount) + require.Contains(t, result.QualityFlags, "retry_backfill_pending") + require.Contains(t, result.Items[0].QualityFlags, "retry_backfill_pending") +} diff --git a/service/channel_select.go b/service/channel_select.go index 2882290e..87bf85ec 100644 --- a/service/channel_select.go +++ b/service/channel_select.go @@ -21,6 +21,18 @@ type RetryParam struct { excludedChannelIDs map[int]struct{} } +// SetSelectedRoutingGroup keeps the configured route and the actual group in +// separate context keys. ContextKeyUsingGroup is the effective group used by +// logging/billing; ContextKeyTokenGroup and the route plan retain the token's +// configured routing policy. +func SetSelectedRoutingGroup(c *gin.Context, group string) { + if c == nil || group == "" { + return + } + common.SetContextKey(c, constant.ContextKeyAutoGroup, group) + common.SetContextKey(c, constant.ContextKeyUsingGroup, group) +} + func (p *RetryParam) GetRetry() int { if p.Retry == nil { return 0 @@ -148,7 +160,7 @@ func CacheGetRandomSatisfiedChannel(param *RetryParam) (*model.Channel, string, param.SetRetry(0) continue } - common.SetContextKey(param.Ctx, constant.ContextKeyAutoGroup, autoGroup) + SetSelectedRoutingGroup(param.Ctx, autoGroup) selectGroup = autoGroup logger.LogDebug(param.Ctx, "Auto selected group: %s", autoGroup) @@ -229,7 +241,7 @@ func selectChannelFromTokenRoutePlan(param *RetryParam, plan *TokenRoutePlan) (* continue } - common.SetContextKey(param.Ctx, constant.ContextKeyAutoGroup, group) + SetSelectedRoutingGroup(param.Ctx, group) if plan.RetryOnFailure && index < len(plan.OrderedGroups)-1 { common.SetContextKey(param.Ctx, constant.ContextKeyAutoGroupIndex, index+1) common.SetContextKey(param.Ctx, constant.ContextKeyAutoGroupRetryIndex, param.GetRetry()+1) diff --git a/service/channel_select_routing_test.go b/service/channel_select_routing_test.go index fb15fa74..25f3fe37 100644 --- a/service/channel_select_routing_test.go +++ b/service/channel_select_routing_test.go @@ -72,6 +72,7 @@ func TestTokenRoutePlanAdvancesGroupsWithoutResettingRetryBudget(t *testing.T) { require.NotNil(t, first) require.Equal(t, "base", firstGroup) require.Equal(t, 0, param.GetRetry()) + require.Equal(t, "base", common.GetContextKeyString(ctx, constant.ContextKeyUsingGroup)) param.ExcludeChannel(first.Id) param.SetRetry(1) @@ -80,6 +81,7 @@ func TestTokenRoutePlanAdvancesGroupsWithoutResettingRetryBudget(t *testing.T) { require.NotNil(t, second) require.Equal(t, "deluxe", secondGroup) require.Equal(t, 1, param.GetRetry()) + require.Equal(t, "deluxe", common.GetContextKeyString(ctx, constant.ContextKeyUsingGroup)) } func TestTokenRoutePlanDoesNotAdvanceAfterFailureWhenRetryDisabled(t *testing.T) { diff --git a/service/model_performance.go b/service/model_performance.go new file mode 100644 index 00000000..4ba87c90 --- /dev/null +++ b/service/model_performance.go @@ -0,0 +1,323 @@ +package service + +import ( + "context" + "errors" + "fmt" + "strings" + "time" + + "github.com/MAX-API-Next/MAX-API/common" + "github.com/MAX-API-Next/MAX-API/constant" + "github.com/MAX-API-Next/MAX-API/logger" + "github.com/MAX-API-Next/MAX-API/model" + perfmetrics "github.com/MAX-API-Next/MAX-API/pkg/perf_metrics" +) + +var ErrInvalidModelPerformanceQuery = errors.New("invalid model performance query") + +const modelPerformanceDetailHours = 24 + +type ModelPerformanceQuery = model.ModelPerformanceQuery + +type ModelPerformanceTimeRange struct { + StartAt int64 `json:"start_at"` + EndAt int64 `json:"end_at"` + Hours int `json:"hours"` +} + +type ModelPerformanceSummaryStats struct { + ModelCount int `json:"model_count"` + ChannelCount int `json:"channel_count"` + ObservedCount int64 `json:"observed_count"` + ConsumeLogCount int64 `json:"consume_log_count"` + ErrorLogCount int64 `json:"error_log_count"` + ConsumedQuota int64 `json:"consumed_quota"` + RetryLogCount *int64 `json:"retry_log_count"` + LatencySampleCount int64 `json:"latency_sample_count"` + ObservedSuccessRate *float64 `json:"observed_success_rate"` + AvgLoggedLatencyMs *float64 `json:"avg_logged_latency_ms"` + LastObservedAt int64 `json:"last_observed_at"` +} + +type ModelPerformanceItem struct { + ModelName string `json:"model_name"` + ChannelCount int `json:"channel_count"` + ObservedCount int64 `json:"observed_count"` + ConsumeLogCount int64 `json:"consume_log_count"` + ErrorLogCount int64 `json:"error_log_count"` + ConsumedQuota int64 `json:"consumed_quota"` + RetryLogCount *int64 `json:"retry_log_count"` + LatencySampleCount int64 `json:"latency_sample_count"` + ObservedSuccessRate *float64 `json:"observed_success_rate"` + AvgLoggedLatencyMs *float64 `json:"avg_logged_latency_ms"` + AvgTps *float64 `json:"avg_tps"` + LastObservedAt int64 `json:"last_observed_at"` + QualityFlags []string `json:"quality_flags"` +} + +type ModelPerformanceSummary struct { + StorageMode string `json:"storage_mode"` + Source string `json:"source"` + Partial bool `json:"partial"` + QualityFlags []string `json:"quality_flags"` + TimeRange ModelPerformanceTimeRange `json:"time_range"` + Summary ModelPerformanceSummaryStats `json:"summary"` + Items []ModelPerformanceItem `json:"items"` + Throughput ModelPerformanceThroughputInfo `json:"throughput"` + Truncated bool `json:"truncated"` + GeneratedAt int64 `json:"generated_at"` +} + +type ModelPerformanceReader interface { + Summary(context.Context, ModelPerformanceQuery) (ModelPerformanceSummary, error) +} + +type legacyModelPerformanceReader struct{} + +type modelPerformanceThroughputSnapshot struct { + State perfmetrics.CollectionState + Coverage perfmetrics.QueryCoverage + ByModel map[string]float64 +} + +type ModelPerformanceThroughputInfo struct { + CollectionState perfmetrics.CollectionState `json:"collection_state"` + Coverage perfmetrics.QueryCoverage `json:"coverage"` +} + +func (legacyModelPerformanceReader) Summary(ctx context.Context, query ModelPerformanceQuery) (ModelPerformanceSummary, error) { + query, rangeWasClamped, err := normalizeModelPerformanceQuery(query) + if err != nil { + return ModelPerformanceSummary{}, err + } + + legacyResult, err := model.QueryLegacyModelPerformance(ctx, query) + if err != nil { + return ModelPerformanceSummary{}, err + } + rows := legacyResult.Rows + truncated := len(rows) > query.Limit + if truncated { + rows = rows[:query.Limit] + } + throughput := queryModelPerformanceThroughput(ctx, query) + + return buildLegacyModelPerformanceSummary( + query, + rangeWasClamped, + legacyResult, + rows, + truncated, + common.LogConsumeEnabled, + constant.ErrorLogEnabled, + throughput, + time.Now().Unix(), + ), nil +} + +func buildLegacyModelPerformanceSummary( + query ModelPerformanceQuery, + rangeWasClamped bool, + legacyResult model.LegacyModelPerformanceResult, + rows []model.LegacyModelPerformanceRow, + truncated bool, + consumeLogsEnabled bool, + errorLogsEnabled bool, + throughput modelPerformanceThroughputSnapshot, + generatedAt int64, +) ModelPerformanceSummary { + qualityFlags := []string{ + "legacy", + "partial", + "weak_correlation", + "heuristic_outcome", + "coarse_latency", + "non_attempt_logs_excluded", + "incomplete_errors_possible", + } + if rangeWasClamped { + qualityFlags = append(qualityFlags, "time_range_clamped") + } + if truncated { + qualityFlags = append(qualityFlags, "truncated") + } + switch legacyResult.RetryMetricState { + case model.LegacyRetryMetricStatePending: + qualityFlags = append(qualityFlags, "retry_backfill_pending") + case model.LegacyRetryMetricStateUnavailable: + qualityFlags = append(qualityFlags, "retry_metrics_unavailable") + } + if !consumeLogsEnabled { + qualityFlags = append(qualityFlags, "consume_logs_disabled") + } + if !errorLogsEnabled { + qualityFlags = append(qualityFlags, "error_logs_disabled") + } + switch throughput.State { + case perfmetrics.CollectionStatePartial: + qualityFlags = append(qualityFlags, "throughput_partial") + case perfmetrics.CollectionStateDisabled: + qualityFlags = append(qualityFlags, "throughput_collection_disabled") + case perfmetrics.CollectionStateNoSamples: + qualityFlags = append(qualityFlags, "throughput_no_samples") + case perfmetrics.CollectionStateQueryFailed: + qualityFlags = append(qualityFlags, "throughput_query_failed") + } + if throughput.Coverage.Approximate && throughput.State != perfmetrics.CollectionStateNoSamples && throughput.State != perfmetrics.CollectionStateQueryFailed { + qualityFlags = append(qualityFlags, "throughput_window_approximate") + } + + allowSuccessRate := consumeLogsEnabled && errorLogsEnabled + items := make([]ModelPerformanceItem, 0, len(rows)) + for _, row := range rows { + item := ModelPerformanceItem{ + ModelName: row.ModelName, + ChannelCount: int(row.ChannelCount), + ObservedCount: row.ObservedCount, + ConsumeLogCount: row.ConsumeLogCount, + ErrorLogCount: row.ErrorLogCount, + ConsumedQuota: row.ConsumedQuota, + LatencySampleCount: row.LatencySampleCount, + LastObservedAt: row.LastObservedAt, + QualityFlags: append([]string(nil), qualityFlags...), + } + if legacyResult.RetryMetricsAvailable { + item.RetryLogCount = int64Pointer(row.RetryLogCount) + } + if item.LatencySampleCount > 0 { + item.AvgLoggedLatencyMs = floatPointer(float64(row.TotalUseTimeSeconds*1000) / float64(item.LatencySampleCount)) + } + if avgTps, ok := throughput.ByModel[row.ModelName]; ok { + item.AvgTps = floatPointer(avgTps) + } + if allowSuccessRate && item.ObservedCount > 0 { + item.ObservedSuccessRate = floatPointer(float64(item.ConsumeLogCount) / float64(item.ObservedCount) * 100) + } + items = append(items, item) + } + + summary := ModelPerformanceSummaryStats{ + ModelCount: int(legacyResult.Summary.ModelCount), + ChannelCount: int(legacyResult.Summary.ChannelCount), + ObservedCount: legacyResult.Summary.ObservedCount, + ConsumeLogCount: legacyResult.Summary.ConsumeLogCount, + ErrorLogCount: legacyResult.Summary.ErrorLogCount, + ConsumedQuota: legacyResult.Summary.ConsumedQuota, + LatencySampleCount: legacyResult.Summary.LatencySampleCount, + LastObservedAt: legacyResult.Summary.LastObservedAt, + } + if legacyResult.RetryMetricsAvailable { + summary.RetryLogCount = int64Pointer(legacyResult.Summary.RetryLogCount) + } + if allowSuccessRate && summary.ObservedCount > 0 { + summary.ObservedSuccessRate = floatPointer(float64(summary.ConsumeLogCount) / float64(summary.ObservedCount) * 100) + } + if summary.LatencySampleCount > 0 { + summary.AvgLoggedLatencyMs = floatPointer(float64(legacyResult.Summary.TotalUseTimeSeconds*1000) / float64(summary.LatencySampleCount)) + } + + return ModelPerformanceSummary{ + StorageMode: "legacy_log", + Source: "log_db", + Partial: true, + QualityFlags: qualityFlags, + TimeRange: ModelPerformanceTimeRange{ + StartAt: query.StartAt, + EndAt: query.EndAt, + Hours: int((query.EndAt - query.StartAt) / int64(time.Hour/time.Second)), + }, + Summary: summary, + Items: items, + Throughput: ModelPerformanceThroughputInfo{ + CollectionState: throughput.State, + Coverage: throughput.Coverage, + }, + Truncated: truncated, + GeneratedAt: generatedAt, + } +} + +var queryModelPerformanceSummaryRange = perfmetrics.QuerySummaryAllRangeDetailedContext + +func queryModelPerformanceThroughput(ctx context.Context, query ModelPerformanceQuery) modelPerformanceThroughputSnapshot { + groups := []string(nil) + if query.Group != "" { + groups = []string{query.Group} + } + result, err := queryModelPerformanceSummaryRange(ctx, query.StartAt, query.EndAt, groups) + if err != nil { + logger.LogError(ctx, "failed to query model performance throughput: "+err.Error()) + return modelPerformanceThroughputSnapshot{State: perfmetrics.CollectionStateQueryFailed} + } + byModel := make(map[string]float64, len(result.Models)) + for _, summary := range result.Models { + if query.ModelName != "" && summary.ModelName != query.ModelName { + continue + } + byModel[summary.ModelName] = summary.AvgTps + } + return modelPerformanceThroughputSnapshot{ + State: collectionStateForModelFilter(result.CollectionState, query.ModelName, byModel), + Coverage: result.Coverage, + ByModel: byModel, + } +} + +func collectionStateForModelFilter( + state perfmetrics.CollectionState, + modelName string, + byModel map[string]float64, +) perfmetrics.CollectionState { + if modelName == "" { + return state + } + if _, hasSamples := byModel[modelName]; hasSamples { + return state + } + switch state { + case perfmetrics.CollectionStateDisabled: + return perfmetrics.CollectionStateDisabled + case perfmetrics.CollectionStatePartial: + return perfmetrics.CollectionStatePartial + case perfmetrics.CollectionStateQueryFailed: + return perfmetrics.CollectionStateQueryFailed + default: + return perfmetrics.CollectionStateNoSamples + } +} + +var defaultModelPerformanceReader ModelPerformanceReader = legacyModelPerformanceReader{} + +func GetModelPerformance(ctx context.Context, query ModelPerformanceQuery) (ModelPerformanceSummary, error) { + return defaultModelPerformanceReader.Summary(ctx, query) +} + +func GetModelPerformanceDetail(ctx context.Context, modelName string) (perfmetrics.DetailedQueryResult, error) { + modelName = strings.TrimSpace(modelName) + if modelName == "" { + return perfmetrics.DetailedQueryResult{}, fmt.Errorf("%w: model is required", ErrInvalidModelPerformanceQuery) + } + return perfmetrics.QueryDetailedContext(ctx, perfmetrics.QueryParams{ + Model: modelName, + Hours: modelPerformanceDetailHours, + }) +} + +func normalizeModelPerformanceQuery(query ModelPerformanceQuery) (ModelPerformanceQuery, bool, error) { + normalized, rangeWasClamped, err := normalizePerformanceQuery( + query.StartAt, + query.EndAt, + query.Hours, + query.Limit, + ErrInvalidModelPerformanceQuery, + ) + if err != nil { + return query, false, err + } + query.StartAt = normalized.StartAt + query.EndAt = normalized.EndAt + query.Hours = normalized.Hours + query.Limit = normalized.Limit + return query, rangeWasClamped, nil +} diff --git a/service/model_performance_test.go b/service/model_performance_test.go new file mode 100644 index 00000000..fd24e92e --- /dev/null +++ b/service/model_performance_test.go @@ -0,0 +1,212 @@ +package service + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/MAX-API-Next/MAX-API/model" + perfmetrics "github.com/MAX-API-Next/MAX-API/pkg/perf_metrics" + "github.com/stretchr/testify/require" +) + +func TestNormalizeModelPerformanceQueryUsesOneHourDefaultAndClampsWindow(t *testing.T) { + before := time.Now().Unix() + query, clamped, err := normalizeModelPerformanceQuery(ModelPerformanceQuery{}) + after := time.Now().Unix() + + require.NoError(t, err) + require.False(t, clamped) + require.GreaterOrEqual(t, query.EndAt, before) + require.LessOrEqual(t, query.EndAt, after) + require.EqualValues(t, time.Hour/time.Second, query.EndAt-query.StartAt) + + query, clamped, err = normalizeModelPerformanceQuery(ModelPerformanceQuery{Hours: 37}) + require.NoError(t, err) + require.False(t, clamped) + require.EqualValues(t, 37*time.Hour/time.Second, query.EndAt-query.StartAt) + + query, clamped, err = normalizeModelPerformanceQuery(ModelPerformanceQuery{Hours: 999}) + require.NoError(t, err) + require.True(t, clamped) + require.EqualValues(t, maxPerformanceHours*time.Hour/time.Second, query.EndAt-query.StartAt) +} + +func TestNormalizeModelPerformanceQueryRejectsReversedRange(t *testing.T) { + _, _, err := normalizeModelPerformanceQuery(ModelPerformanceQuery{StartAt: 20, EndAt: 10}) + + require.Error(t, err) + require.True(t, errors.Is(err, ErrInvalidModelPerformanceQuery)) +} + +func TestBuildLegacyModelPerformanceSummaryPreservesGlobalSummaryWhenRowsTruncated(t *testing.T) { + result := buildLegacyModelPerformanceSummary( + ModelPerformanceQuery{StartAt: 1_700_000_000, EndAt: 1_700_003_600, Limit: 1}, + false, + model.LegacyModelPerformanceResult{ + Rows: []model.LegacyModelPerformanceRow{ + {ModelName: "alpha"}, + {ModelName: "beta"}, + }, + Summary: model.LegacyModelPerformanceSummary{ + ModelCount: 2, + ChannelCount: 3, + ObservedCount: 5, + ConsumeLogCount: 3, + ErrorLogCount: 2, + ConsumedQuota: 900, + RetryLogCount: 1, + LatencySampleCount: 4, + TotalUseTimeSeconds: 3, + LastObservedAt: 1_700_003_500, + }, + RetryMetricsAvailable: true, + RetryMetricState: model.LegacyRetryMetricStateAvailable, + }, + []model.LegacyModelPerformanceRow{{ + ModelName: "alpha", + ChannelCount: 2, + ObservedCount: 3, + ConsumeLogCount: 2, + ErrorLogCount: 1, + ConsumedQuota: 500, + RetryLogCount: 1, + LatencySampleCount: 2, + TotalUseTimeSeconds: 1, + LastObservedAt: 1_700_003_400, + }}, + true, + true, + true, + modelPerformanceThroughputSnapshot{ + State: perfmetrics.CollectionStateAvailable, + ByModel: map[string]float64{"alpha": 42.5}, + }, + 1_700_003_600, + ) + + require.True(t, result.Truncated) + require.Len(t, result.Items, 1) + require.Contains(t, result.QualityFlags, "truncated") + require.Equal(t, 2, result.Summary.ModelCount) + require.Equal(t, 3, result.Summary.ChannelCount) + require.EqualValues(t, 5, result.Summary.ObservedCount) + require.EqualValues(t, 900, result.Summary.ConsumedQuota) + require.NotNil(t, result.Summary.ObservedSuccessRate) + require.InDelta(t, 60, *result.Summary.ObservedSuccessRate, 0.001) + require.NotNil(t, result.Summary.AvgLoggedLatencyMs) + require.InDelta(t, 750, *result.Summary.AvgLoggedLatencyMs, 0.001) + require.NotNil(t, result.Items[0].RetryLogCount) + require.EqualValues(t, 500, result.Items[0].ConsumedQuota) + require.EqualValues(t, 1, *result.Items[0].RetryLogCount) + require.NotNil(t, result.Items[0].AvgTps) + require.InDelta(t, 42.5, *result.Items[0].AvgTps, 0.001) +} + +func TestBuildLegacyModelPerformanceSummaryDoesNotInventMetricsWhenUnavailable(t *testing.T) { + result := buildLegacyModelPerformanceSummary( + ModelPerformanceQuery{StartAt: 1, EndAt: 3_601, Limit: 10}, + false, + model.LegacyModelPerformanceResult{ + Summary: model.LegacyModelPerformanceSummary{ObservedCount: 1, ConsumeLogCount: 1}, + RetryMetricState: model.LegacyRetryMetricStatePending, + }, + []model.LegacyModelPerformanceRow{{ + ModelName: "alpha", ObservedCount: 1, ConsumeLogCount: 1, + }}, + false, + true, + true, + modelPerformanceThroughputSnapshot{State: perfmetrics.CollectionStateQueryFailed}, + 3_601, + ) + + require.Nil(t, result.Summary.RetryLogCount) + require.Nil(t, result.Items[0].RetryLogCount) + require.Nil(t, result.Items[0].AvgLoggedLatencyMs) + require.Nil(t, result.Items[0].AvgTps) + require.Contains(t, result.QualityFlags, "retry_backfill_pending") + require.Contains(t, result.QualityFlags, "throughput_query_failed") +} + +func TestBuildLegacyModelPerformanceSummaryDistinguishesThroughputStates(t *testing.T) { + tests := []struct { + name string + state perfmetrics.CollectionState + flag string + }{ + {name: "collection disabled", state: perfmetrics.CollectionStateDisabled, flag: "throughput_collection_disabled"}, + {name: "no samples", state: perfmetrics.CollectionStateNoSamples, flag: "throughput_no_samples"}, + {name: "query failed", state: perfmetrics.CollectionStateQueryFailed, flag: "throughput_query_failed"}, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + result := buildLegacyModelPerformanceSummary( + ModelPerformanceQuery{StartAt: 1, EndAt: 3_601, Limit: 10}, + false, + model.LegacyModelPerformanceResult{}, + nil, + false, + true, + true, + modelPerformanceThroughputSnapshot{State: test.state}, + 3_601, + ) + + require.Contains(t, result.QualityFlags, test.flag) + require.Equal(t, test.state, result.Throughput.CollectionState) + }) + } +} + +func TestQueryModelPerformanceThroughputMarksQueryFailure(t *testing.T) { + originalQuery := queryModelPerformanceSummaryRange + queryModelPerformanceSummaryRange = func(context.Context, int64, int64, []string) (perfmetrics.DetailedSummaryAllResult, error) { + return perfmetrics.DetailedSummaryAllResult{}, errors.New("database unavailable") + } + t.Cleanup(func() { + queryModelPerformanceSummaryRange = originalQuery + }) + + snapshot := queryModelPerformanceThroughput(context.Background(), ModelPerformanceQuery{StartAt: 1, EndAt: 3_601}) + + require.Equal(t, perfmetrics.CollectionStateQueryFailed, snapshot.State) + require.Empty(t, snapshot.ByModel) +} + +func TestQueryModelPerformanceThroughputScopesCollectionStateToModelFilter(t *testing.T) { + originalQuery := queryModelPerformanceSummaryRange + queryModelPerformanceSummaryRange = func(context.Context, int64, int64, []string) (perfmetrics.DetailedSummaryAllResult, error) { + return perfmetrics.DetailedSummaryAllResult{ + Models: []perfmetrics.ModelSummary{{ModelName: "beta", AvgTps: 12.5}}, + CollectionState: perfmetrics.CollectionStateAvailable, + }, nil + } + t.Cleanup(func() { + queryModelPerformanceSummaryRange = originalQuery + }) + + snapshot := queryModelPerformanceThroughput(context.Background(), ModelPerformanceQuery{ + StartAt: 1, + EndAt: 3_601, + ModelName: "alpha", + }) + + require.Equal(t, perfmetrics.CollectionStateNoSamples, snapshot.State) + require.Empty(t, snapshot.ByModel) + + queryModelPerformanceSummaryRange = func(context.Context, int64, int64, []string) (perfmetrics.DetailedSummaryAllResult, error) { + return perfmetrics.DetailedSummaryAllResult{ + Models: []perfmetrics.ModelSummary{{ModelName: "beta", AvgTps: 12.5}}, + CollectionState: perfmetrics.CollectionStatePartial, + }, nil + } + snapshot = queryModelPerformanceThroughput(context.Background(), ModelPerformanceQuery{ + StartAt: 1, + EndAt: 3_601, + ModelName: "alpha", + }) + require.Equal(t, perfmetrics.CollectionStatePartial, snapshot.State) +} diff --git a/service/openaicompat/chat_to_responses.go b/service/openaicompat/chat_to_responses.go index 171353ea..5b6bb3b1 100644 --- a/service/openaicompat/chat_to_responses.go +++ b/service/openaicompat/chat_to_responses.go @@ -371,6 +371,13 @@ func ChatCompletionsRequestToResponsesRequest(req *dto.GeneralOpenAIRequest) (*d if req.TopP != nil { topP = common.GetPointer(lo.FromPtr(req.TopP)) } + var promptCacheKeyRaw json.RawMessage + if req.PromptCacheKey != "" { + promptCacheKeyRaw, err = common.Marshal(req.PromptCacheKey) + if err != nil { + return nil, fmt.Errorf("marshal prompt_cache_key: %w", err) + } + } out := &dto.OpenAIResponsesRequest{ Model: req.Model, @@ -382,10 +389,13 @@ func ChatCompletionsRequestToResponsesRequest(req *dto.GeneralOpenAIRequest) (*d ToolChoice: toolChoiceRaw, Tools: toolsRaw, TopP: topP, + FrequencyPenalty: req.FrequencyPenalty, + PresencePenalty: req.PresencePenalty, User: req.User, ParallelToolCalls: parallelToolCallsRaw, Store: req.Store, Metadata: req.Metadata, + PromptCacheKey: promptCacheKeyRaw, EnableThinking: req.EnableThinking, ThinkingBudget: req.ThinkingBudget, } diff --git a/service/openaicompat/responses_compat_test.go b/service/openaicompat/responses_compat_test.go index c3bbfb3a..58b72316 100644 --- a/service/openaicompat/responses_compat_test.go +++ b/service/openaicompat/responses_compat_test.go @@ -57,6 +57,29 @@ func TestChatCompletionsRequestToResponsesRequestPreservesQwenThinkingBudget(t * assert.Equal(t, "0", string(got.ThinkingBudget)) } +func TestChatCompletionsRequestToResponsesPreservesCacheKeyAndExplicitZeroPenalties(t *testing.T) { + zero := 0.0 + got, err := ChatCompletionsRequestToResponsesRequest(&dto.GeneralOpenAIRequest{ + Model: "gpt-test", + Messages: []dto.Message{{Role: "user", Content: "hello"}}, + PromptCacheKey: "cache-key", + FrequencyPenalty: &zero, + PresencePenalty: &zero, + }) + require.NoError(t, err) + payload, err := common.Marshal(got) + require.NoError(t, err) + require.JSONEq(t, `{"model":"gpt-test","input":[{"content":"hello","role":"user"}],"prompt_cache_key":"cache-key","frequency_penalty":0,"presence_penalty":0}`, string(payload)) + + without, err := ChatCompletionsRequestToResponsesRequest(&dto.GeneralOpenAIRequest{Model: "gpt-test"}) + require.NoError(t, err) + payload, err = common.Marshal(without) + require.NoError(t, err) + require.NotContains(t, string(payload), "frequency_penalty") + require.NotContains(t, string(payload), "presence_penalty") + require.NotContains(t, string(payload), "prompt_cache_key") +} + func TestResponsesRequestToChatCompletionsRequestPreservesQwenThinkingBudget(t *testing.T) { got, err := ResponsesRequestToChatCompletionsRequest(&dto.OpenAIResponsesRequest{ Model: "qwen-plus", @@ -70,6 +93,22 @@ func TestResponsesRequestToChatCompletionsRequestPreservesQwenThinkingBudget(t * assert.Equal(t, "0", string(got.ThinkingBudget)) } +func TestResponsesRequestToChatCompletionsRequestPreservesPenalties(t *testing.T) { + frequencyPenalty := 0.0 + presencePenalty := 1.5 + got, err := ResponsesRequestToChatCompletionsRequest(&dto.OpenAIResponsesRequest{ + Model: "gpt-test", + Input: mustCompatRawMessage(t, "hello"), + FrequencyPenalty: &frequencyPenalty, + PresencePenalty: &presencePenalty, + }) + require.NoError(t, err) + require.NotNil(t, got.FrequencyPenalty) + require.NotNil(t, got.PresencePenalty) + assert.Equal(t, 0.0, *got.FrequencyPenalty) + assert.Equal(t, 1.5, *got.PresencePenalty) +} + func TestResponsesRequestToChatCompletionsRequestFunctionCallConversation(t *testing.T) { got, err := ResponsesRequestToChatCompletionsRequest(&dto.OpenAIResponsesRequest{ Model: "gpt-test", diff --git a/service/openaicompat/responses_request_to_chat.go b/service/openaicompat/responses_request_to_chat.go index 3c7d91f7..ab024331 100644 --- a/service/openaicompat/responses_request_to_chat.go +++ b/service/openaicompat/responses_request_to_chat.go @@ -65,6 +65,8 @@ func ResponsesRequestToChatCompletionsRequest(req *dto.OpenAIResponsesRequest) ( EnableThinking: req.EnableThinking, ThinkingBudget: req.ThinkingBudget, } + out.FrequencyPenalty = req.FrequencyPenalty + out.PresencePenalty = req.PresencePenalty if req.Reasoning != nil { out.ReasoningEffort = req.Reasoning.Effort } diff --git a/service/performance_query.go b/service/performance_query.go new file mode 100644 index 00000000..caeee4b5 --- /dev/null +++ b/service/performance_query.go @@ -0,0 +1,67 @@ +package service + +import ( + "errors" + "fmt" + "time" +) + +const ( + defaultPerformanceHours = 1 + maxPerformanceHours = 168 + defaultPerformanceLimit = 100 + maxPerformanceLimit = 200 +) + +type normalizedPerformanceQuery struct { + StartAt int64 + EndAt int64 + Hours int + Limit int +} + +func normalizePerformanceQuery( + startAt int64, + endAt int64, + hours int, + limit int, + invalidQueryError error, +) (normalizedPerformanceQuery, bool, error) { + if invalidQueryError == nil { + invalidQueryError = errors.New("invalid performance query") + } + if endAt <= 0 { + endAt = time.Now().Unix() + } + rangeWasClamped := false + if startAt <= 0 { + if hours <= 0 { + hours = defaultPerformanceHours + } + if hours > maxPerformanceHours { + hours = maxPerformanceHours + rangeWasClamped = true + } + startAt = endAt - int64(hours)*int64(time.Hour/time.Second) + } + if endAt <= startAt { + return normalizedPerformanceQuery{}, false, fmt.Errorf("%w: end must be greater than start", invalidQueryError) + } + if limit <= 0 { + limit = defaultPerformanceLimit + } + if limit > maxPerformanceLimit { + limit = maxPerformanceLimit + } + maxWindow := int64(maxPerformanceHours) * int64(time.Hour/time.Second) + if endAt-startAt > maxWindow { + startAt = endAt - maxWindow + rangeWasClamped = true + } + return normalizedPerformanceQuery{ + StartAt: startAt, + EndAt: endAt, + Hours: hours, + Limit: limit, + }, rangeWasClamped, nil +} diff --git a/service/smart_ops_alert.go b/service/smart_ops_alert.go new file mode 100644 index 00000000..3f61dc80 --- /dev/null +++ b/service/smart_ops_alert.go @@ -0,0 +1,534 @@ +package service + +import ( + "context" + "fmt" + "hash/fnv" + "sort" + "sync" + "time" + + "github.com/MAX-API-Next/MAX-API/common" + "github.com/MAX-API-Next/MAX-API/dto" + "github.com/MAX-API-Next/MAX-API/model" +) + +const ( + smartOpsAlertStatusFiring = "firing" + smartOpsAlertStatusResolved = "resolved" + smartOpsAlertSeverityWarning = "warning" + smartOpsAlertSampleInterval = 5 * time.Second + smartOpsAlertRequiredSamples = 2 + smartOpsAlertQueueSize = 24 + smartOpsAlertWorkerCount = 3 +) + +var smartOpsAlertRetryDelays = []time.Duration{ + 5 * time.Second, + 15 * time.Second, + 45 * time.Second, +} + +// SmartOpsAlert is the read-only incident projection exposed to administrators. +// It intentionally contains only resource metadata and never includes secrets, +// request bodies, channel keys, or billing data. +type SmartOpsAlert struct { + Key string `json:"key"` + Status string `json:"status"` + Severity string `json:"severity"` + Component string `json:"component"` + Node string `json:"node,omitempty"` + CurrentValue float64 `json:"current_value"` + Threshold float64 `json:"threshold"` + ObservedAt time.Time `json:"observed_at"` + Message string `json:"message"` +} + +type smartOpsAlertEvent struct { + Key string + Status string + CurrentValue float64 + Threshold float64 + ObservedAt time.Time +} + +type smartOpsAlertState struct { + active bool + consecutive int + lastValue float64 + lastObserved time.Time +} + +func (state *smartOpsAlertState) suppress(observedAt time.Time) { + state.active = false + state.consecutive = 0 + state.lastValue = 0 + state.lastObserved = observedAt +} + +func (state *smartOpsAlertState) observe(key string, value, threshold float64, observedAt time.Time, requiredSamples int) *smartOpsAlertEvent { + if threshold <= 0 { + state.suppress(observedAt) + return nil + } + if observedAt.IsZero() || !observedAt.After(state.lastObserved) { + return nil + } + + state.lastValue = value + state.lastObserved = observedAt + if value <= threshold { + state.consecutive = 0 + if !state.active { + return nil + } + state.active = false + return &smartOpsAlertEvent{Key: key, Status: smartOpsAlertStatusResolved, CurrentValue: value, Threshold: threshold, ObservedAt: observedAt} + } + + state.consecutive++ + if state.active || state.consecutive < requiredSamples { + return nil + } + state.active = true + return &smartOpsAlertEvent{Key: key, Status: smartOpsAlertStatusFiring, CurrentValue: value, Threshold: threshold, ObservedAt: observedAt} +} + +type smartOpsAlertDefinition struct { + key string + component string + label string + value func(common.SystemStatus) float64 + valid func(common.SystemStatus) bool + threshold func(common.PerformanceMonitorConfig) float64 +} + +var smartOpsAlertDefinitions = []smartOpsAlertDefinition{ + { + key: "system_cpu", + component: "system", + label: "CPU 使用率", + value: func(status common.SystemStatus) float64 { return status.CPUUsage }, + valid: func(status common.SystemStatus) bool { return status.CPUValid }, + threshold: func(config common.PerformanceMonitorConfig) float64 { return float64(config.CPUThreshold) }, + }, + { + key: "system_memory", + component: "system", + label: "内存使用率", + value: func(status common.SystemStatus) float64 { return status.MemoryUsage }, + valid: func(status common.SystemStatus) bool { return status.MemoryValid }, + threshold: func(config common.PerformanceMonitorConfig) float64 { return float64(config.MemoryThreshold) }, + }, + { + key: "system_disk", + component: "system", + label: "磁盘使用率", + value: func(status common.SystemStatus) float64 { return status.DiskUsage }, + valid: func(status common.SystemStatus) bool { return status.DiskValid }, + threshold: func(config common.PerformanceMonitorConfig) float64 { return float64(config.DiskThreshold) }, + }, +} + +var smartOpsAlertMonitor struct { + sync.Mutex + cancel context.CancelFunc + done chan struct{} + states map[string]*smartOpsAlertState + active map[string]SmartOpsAlert +} + +var smartOpsAlertNotificationSender = enqueueSmartOpsAlertNotification + +// StartSmartOpsAlertMonitor starts the in-process detector. It is deliberately +// separate from the resource sampler so the existing sampler remains cheap and +// the detector can be disabled or stopped independently during shutdown. +func StartSmartOpsAlertMonitor() { + smartOpsAlertMonitor.Lock() + defer smartOpsAlertMonitor.Unlock() + if smartOpsAlertMonitor.cancel != nil { + return + } + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + smartOpsAlertMonitor.cancel = cancel + smartOpsAlertMonitor.done = done + smartOpsAlertMonitor.states = make(map[string]*smartOpsAlertState) + smartOpsAlertMonitor.active = make(map[string]SmartOpsAlert) + go func() { + defer close(done) + runSmartOpsAlertMonitor(ctx) + }() +} + +func runSmartOpsAlertMonitor(ctx context.Context) { + ticker := time.NewTicker(smartOpsAlertSampleInterval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case now := <-ticker.C: + evaluateSmartOpsSystemAlerts(common.GetSystemStatus(), common.GetPerformanceMonitorConfig(), now) + } + } +} + +// StopSmartOpsAlertMonitor stops the detector without affecting request +// handling or the resource sampler. +func StopSmartOpsAlertMonitor(ctx context.Context) error { + smartOpsAlertMonitor.Lock() + cancel := smartOpsAlertMonitor.cancel + done := smartOpsAlertMonitor.done + smartOpsAlertMonitor.Unlock() + if cancel == nil || done == nil { + return nil + } + + cancel() + select { + case <-done: + smartOpsAlertMonitor.Lock() + if smartOpsAlertMonitor.done == done { + smartOpsAlertMonitor.cancel = nil + smartOpsAlertMonitor.done = nil + } + smartOpsAlertMonitor.Unlock() + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +func evaluateSmartOpsSystemAlerts(status common.SystemStatus, config common.PerformanceMonitorConfig, _ time.Time) { + if !config.Enabled { + smartOpsAlertMonitor.Lock() + for _, state := range smartOpsAlertMonitor.states { + state.suppress(status.ObservedAt) + } + clear(smartOpsAlertMonitor.active) + smartOpsAlertMonitor.Unlock() + return + } + + var events []SmartOpsAlert + smartOpsAlertMonitor.Lock() + if smartOpsAlertMonitor.states == nil { + smartOpsAlertMonitor.states = make(map[string]*smartOpsAlertState) + } + if smartOpsAlertMonitor.active == nil { + smartOpsAlertMonitor.active = make(map[string]SmartOpsAlert) + } + for _, definition := range smartOpsAlertDefinitions { + state := smartOpsAlertMonitor.states[definition.key] + if state == nil { + state = &smartOpsAlertState{} + smartOpsAlertMonitor.states[definition.key] = state + } + threshold := definition.threshold(config) + if threshold <= 0 { + state.suppress(status.ObservedAt) + delete(smartOpsAlertMonitor.active, definition.key) + continue + } + if !definition.valid(status) || status.ObservedAt.IsZero() || !status.ObservedAt.After(state.lastObserved) { + continue + } + value := definition.value(status) + event := state.observe( + definition.key, + value, + threshold, + status.ObservedAt, + smartOpsAlertRequiredSamples, + ) + if event == nil { + if state.active { + alert := smartOpsAlertMonitor.active[definition.key] + alert.CurrentValue = value + alert.Threshold = threshold + alert.ObservedAt = status.ObservedAt + alert.Message = formatSmartOpsAlertMessage(definition.label, &smartOpsAlertEvent{ + Status: smartOpsAlertStatusFiring, + CurrentValue: alert.CurrentValue, + Threshold: alert.Threshold, + }) + smartOpsAlertMonitor.active[definition.key] = alert + } + continue + } + alert := SmartOpsAlert{ + Key: event.Key, + Status: event.Status, + Severity: smartOpsAlertSeverityWarning, + Component: definition.component, + Node: smartOpsAlertNodeName(), + CurrentValue: event.CurrentValue, + Threshold: event.Threshold, + ObservedAt: event.ObservedAt, + Message: formatSmartOpsAlertMessage(definition.label, event), + } + if event.Status == smartOpsAlertStatusFiring { + smartOpsAlertMonitor.active[event.Key] = alert + } else { + delete(smartOpsAlertMonitor.active, event.Key) + } + events = append(events, alert) + } + smartOpsAlertMonitor.Unlock() + + for _, alert := range events { + smartOpsAlertNotificationSender(alert) + } +} + +func formatSmartOpsAlertMessage(label string, event *smartOpsAlertEvent) string { + if event.Status == smartOpsAlertStatusResolved { + return fmt.Sprintf("%s 已恢复:当前 %.1f%%,阈值 %.1f%%", label, event.CurrentValue, event.Threshold) + } + return fmt.Sprintf("%s 超过阈值:当前 %.1f%%,阈值 %.1f%%。请检查智能运维中心", label, event.CurrentValue, event.Threshold) +} + +func smartOpsAlertNodeName() string { + identity := common.GetNodeIdentity() + if identity.Name != "" { + return identity.Name + } + return "unknown" +} + +// GetSmartOpsAlerts returns the currently firing alerts for the administrator +// cockpit. It is a process-local projection and may be empty immediately after +// restart until the sampler produces two consecutive samples. +func GetSmartOpsAlerts() []SmartOpsAlert { + smartOpsAlertMonitor.Lock() + defer smartOpsAlertMonitor.Unlock() + alerts := make([]SmartOpsAlert, 0, len(smartOpsAlertMonitor.active)) + for _, alert := range smartOpsAlertMonitor.active { + alerts = append(alerts, alert) + } + sort.Slice(alerts, func(i, j int) bool { return alerts[i].Key < alerts[j].Key }) + return alerts +} + +type smartOpsAlertRecipient struct { + ID int + Email string + Setting dto.UserSetting +} + +type smartOpsAlertNotificationPool struct { + queues []chan SmartOpsAlert + deliver func(SmartOpsAlert) error + closeOnce sync.Once + waitGroup sync.WaitGroup +} + +func newSmartOpsAlertNotificationPool(workerCount, queueSize int, deliver func(SmartOpsAlert) error) *smartOpsAlertNotificationPool { + if workerCount < 1 { + workerCount = 1 + } + if queueSize < workerCount { + queueSize = workerCount + } + + pool := &smartOpsAlertNotificationPool{ + queues: make([]chan SmartOpsAlert, workerCount), + deliver: deliver, + } + baseCapacity := queueSize / workerCount + extraCapacity := queueSize % workerCount + for worker := range workerCount { + capacity := baseCapacity + if worker < extraCapacity { + capacity++ + } + pool.queues[worker] = make(chan SmartOpsAlert, capacity) + pool.waitGroup.Add(1) + go pool.runWorker(pool.queues[worker]) + } + return pool +} + +func (pool *smartOpsAlertNotificationPool) runWorker(queue <-chan SmartOpsAlert) { + defer pool.waitGroup.Done() + for alert := range queue { + if err := pool.deliver(alert); err != nil { + common.SysLog(fmt.Sprintf("failed to deliver smart ops alert notification: %s", err.Error())) + } + } +} + +func (pool *smartOpsAlertNotificationPool) workerIndex(alert SmartOpsAlert) int { + hasher := fnv.New32a() + _, _ = hasher.Write([]byte(alert.Node)) + _, _ = hasher.Write([]byte{0}) + _, _ = hasher.Write([]byte(alert.Key)) + return int(hasher.Sum32() % uint32(len(pool.queues))) +} + +func (pool *smartOpsAlertNotificationPool) enqueue(alert SmartOpsAlert) bool { + queue := pool.queues[pool.workerIndex(alert)] + select { + case queue <- alert: + return true + default: + return false + } +} + +func (pool *smartOpsAlertNotificationPool) close() { + pool.closeOnce.Do(func() { + for _, queue := range pool.queues { + close(queue) + } + pool.waitGroup.Wait() + }) +} + +var smartOpsAlertNotificationQueue struct { + sync.Once + pool *smartOpsAlertNotificationPool +} + +var smartOpsAlertLoadRecipients = loadSmartOpsAlertRecipients +var smartOpsAlertCheckNotificationLimit = CheckNotificationLimit +var smartOpsAlertSendToRecipient = func(recipient smartOpsAlertRecipient, data dto.Notify) error { + return sendUserNotification(recipient.ID, recipient.Email, recipient.Setting, data) +} + +func enqueueSmartOpsAlertNotification(alert SmartOpsAlert) { + smartOpsAlertNotificationQueue.Do(func() { + smartOpsAlertNotificationQueue.pool = newSmartOpsAlertNotificationPool( + smartOpsAlertWorkerCount, + smartOpsAlertQueueSize, + deliverSmartOpsAlertNotification, + ) + }) + + if !smartOpsAlertNotificationQueue.pool.enqueue(alert) { + common.SysLog(fmt.Sprintf("smart ops alert notification queue is full, dropping %s for node %s", alert.Key, alert.Node)) + } +} + +func smartOpsAlertNotification(alert SmartOpsAlert) dto.Notify { + statusText := "异常" + if alert.Status == smartOpsAlertStatusResolved { + statusText = "恢复" + } + subject := fmt.Sprintf("智能运维告警:%s", statusText) + content := fmt.Sprintf("节点:%s\n组件:%s\n%s\n时间:%s\n查看:/smart-ops/alerts\n当前告警接口:/api/smart-ops/alerts", alert.Node, alert.Component, alert.Message, alert.ObservedAt.Format(time.RFC3339)) + return dto.NewNotify(smartOpsAlertNotifyType(alert), subject, content, nil) +} + +func smartOpsAlertNotifyType(alert SmartOpsAlert) string { + return fmt.Sprintf("%s:%s:%s:%s", dto.NotifyTypeSmartOpsAlert, alert.Node, alert.Key, alert.Status) +} + +func deliverSmartOpsAlertNotification(alert SmartOpsAlert) error { + var recipients []smartOpsAlertRecipient + if err := retrySmartOpsAlertOperation(func() error { + var err error + recipients, err = smartOpsAlertLoadRecipients() + return err + }); err != nil { + return fmt.Errorf("failed to load administrator recipients: %w", err) + } + + notification := smartOpsAlertNotification(alert) + var firstErr error + for _, recipient := range recipients { + allowed, err := checkSmartOpsAlertNotificationLimit(recipient.ID, notification.Type) + if err != nil { + if firstErr == nil { + firstErr = err + } + common.SysLog(fmt.Sprintf("failed to reserve notification slot for admin user %d: %s", recipient.ID, err.Error())) + continue + } + if !allowed { + err = fmt.Errorf("notification limit exceeded for admin user %d", recipient.ID) + if firstErr == nil { + firstErr = err + } + common.SysLog(err.Error()) + continue + } + if err := retrySmartOpsAlertOperation(func() error { + return smartOpsAlertSendToRecipient(recipient, notification) + }); err != nil { + if firstErr == nil { + firstErr = err + } + common.SysLog(fmt.Sprintf("failed to notify admin user %d for smart ops alert after retries: %s", recipient.ID, err.Error())) + } + } + return firstErr +} + +func retrySmartOpsAlertOperation(operation func() error) error { + var err error + for attempt := 0; attempt <= len(smartOpsAlertRetryDelays); attempt++ { + if err = operation(); err == nil { + return nil + } + if attempt < len(smartOpsAlertRetryDelays) { + time.Sleep(smartOpsAlertRetryDelays[attempt]) + } + } + return err +} + +func checkSmartOpsAlertNotificationLimit(userID int, notifyType string) (bool, error) { + var allowed bool + err := retrySmartOpsAlertOperation(func() error { + var err error + allowed, err = smartOpsAlertCheckNotificationLimit(userID, notifyType) + return err + }) + return allowed, err +} + +func loadSmartOpsAlertRecipients() ([]smartOpsAlertRecipient, error) { + if model.DB == nil { + return nil, fmt.Errorf("database is not initialized") + } + + var users []model.User + if err := model.DB. + Select("id", "email", "role", "status", "setting"). + Where("status = ? AND role >= ?", common.UserStatusEnabled, common.RoleAdminUser). + Find(&users).Error; err != nil { + return nil, fmt.Errorf("failed to query smart ops notification users: %w", err) + } + + recipients := make([]smartOpsAlertRecipient, 0, len(users)) + for _, user := range users { + recipients = append(recipients, smartOpsAlertRecipient{ + ID: user.Id, + Email: user.Email, + Setting: user.GetSetting(), + }) + } + return recipients, nil +} + +// NotifyAdminUsers broadcasts an operational alert to enabled administrators +// using each administrator's existing notification method configuration. +func NotifyAdminUsers(data dto.Notify) error { + recipients, err := loadSmartOpsAlertRecipients() + if err != nil { + return err + } + + var firstErr error + for _, recipient := range recipients { + if err := NotifyUser(recipient.ID, recipient.Email, recipient.Setting, data); err != nil { + if firstErr == nil { + firstErr = err + } + common.SysLog(fmt.Sprintf("failed to notify admin user %d for smart ops alert: %s", recipient.ID, err.Error())) + } + } + return firstErr +} diff --git a/service/smart_ops_alert_test.go b/service/smart_ops_alert_test.go new file mode 100644 index 00000000..3e2ba9a5 --- /dev/null +++ b/service/smart_ops_alert_test.go @@ -0,0 +1,277 @@ +package service + +import ( + "fmt" + "testing" + "time" + + "github.com/MAX-API-Next/MAX-API/common" + "github.com/MAX-API-Next/MAX-API/dto" + "github.com/stretchr/testify/require" +) + +func TestSmartOpsAlertRequiresConsecutiveSamplesAndNotifiesRecovery(t *testing.T) { + state := smartOpsAlertState{} + firstAt := time.Unix(100, 0) + + event := state.observe("system_cpu", 95, 90, firstAt, 2) + require.Nil(t, event, "a single threshold breach must not page an administrator") + require.False(t, state.active) + + event = state.observe("system_cpu", 96, 90, firstAt.Add(5*time.Second), 2) + require.NotNil(t, event) + require.Equal(t, smartOpsAlertStatusFiring, event.Status) + require.Equal(t, "system_cpu", event.Key) + require.True(t, state.active) + + event = state.observe("system_cpu", 97, 90, firstAt.Add(10*time.Second), 2) + require.Nil(t, event, "an ongoing incident must be deduplicated") + + event = state.observe("system_cpu", 70, 90, firstAt.Add(15*time.Second), 2) + require.NotNil(t, event) + require.Equal(t, smartOpsAlertStatusResolved, event.Status) + require.False(t, state.active) +} + +func TestSmartOpsAlertDoesNotCountTheSameSampleTwice(t *testing.T) { + state := smartOpsAlertState{} + observedAt := time.Unix(150, 0) + + require.Nil(t, state.observe("system_cpu", 95, 90, observedAt, 2)) + event := state.observe("system_cpu", 95, 90, observedAt, 2) + + require.Nil(t, event, "the detector must not turn one sampled value into two consecutive samples") + require.False(t, state.active) + require.Equal(t, 1, state.consecutive) +} + +func TestDisablingSmartOpsThresholdSuppressesWithoutRecoveryNotification(t *testing.T) { + state := smartOpsAlertState{active: true, consecutive: 2} + disabledAt := time.Unix(175, 0) + + event := state.observe("system_cpu", 95, 0, disabledAt, 2) + + require.Nil(t, event, "disabling a detector is not a resource recovery") + require.False(t, state.active) + require.Zero(t, state.consecutive) + require.Equal(t, disabledAt, state.lastObserved, "re-enabling must not count a sample captured before the detector was disabled") +} + +func TestEvaluateSmartOpsSystemAlertsNotifiesOnlyOnTransitions(t *testing.T) { + originalSender := smartOpsAlertNotificationSender + t.Cleanup(func() { smartOpsAlertNotificationSender = originalSender }) + + smartOpsAlertMonitor.Lock() + smartOpsAlertMonitor.states = nil + smartOpsAlertMonitor.active = nil + smartOpsAlertMonitor.Unlock() + + var events []SmartOpsAlert + smartOpsAlertNotificationSender = func(alert SmartOpsAlert) { + events = append(events, alert) + } + + config := common.PerformanceMonitorConfig{ + Enabled: true, + CPUThreshold: 90, + MemoryThreshold: 0, + DiskThreshold: 0, + } + base := time.Unix(200, 0) + status := common.SystemStatus{CPUUsage: 95, CPUValid: true, ObservedAt: base} + + evaluateSmartOpsSystemAlerts(status, config, base) + status.ObservedAt = base.Add(5 * time.Second) + evaluateSmartOpsSystemAlerts(status, config, base.Add(5*time.Second)) + status.ObservedAt = base.Add(10 * time.Second) + evaluateSmartOpsSystemAlerts(status, config, base.Add(10*time.Second)) + status.CPUUsage = 70 + status.ObservedAt = base.Add(15 * time.Second) + evaluateSmartOpsSystemAlerts(status, config, base.Add(15*time.Second)) + + require.Len(t, events, 2) + require.Equal(t, smartOpsAlertStatusFiring, events[0].Status) + require.Equal(t, smartOpsAlertStatusResolved, events[1].Status) +} + +func TestEvaluateSmartOpsSystemAlertsDoesNotResolveFromInvalidProbe(t *testing.T) { + originalSender := smartOpsAlertNotificationSender + t.Cleanup(func() { smartOpsAlertNotificationSender = originalSender }) + + smartOpsAlertMonitor.Lock() + smartOpsAlertMonitor.states = nil + smartOpsAlertMonitor.active = nil + smartOpsAlertMonitor.Unlock() + + var events []SmartOpsAlert + smartOpsAlertNotificationSender = func(alert SmartOpsAlert) { + events = append(events, alert) + } + + config := common.PerformanceMonitorConfig{Enabled: true, CPUThreshold: 90} + base := time.Unix(300, 0) + status := common.SystemStatus{CPUUsage: 95, CPUValid: true, ObservedAt: base} + evaluateSmartOpsSystemAlerts(status, config, base) + status.ObservedAt = base.Add(5 * time.Second) + evaluateSmartOpsSystemAlerts(status, config, base.Add(5*time.Second)) + + status = common.SystemStatus{ObservedAt: base.Add(10 * time.Second)} + evaluateSmartOpsSystemAlerts(status, config, base.Add(10*time.Second)) + + require.Len(t, events, 1) + require.Equal(t, smartOpsAlertStatusFiring, events[0].Status) + require.Len(t, GetSmartOpsAlerts(), 1, "an unknown sample must preserve the active incident") +} + +func TestSmartOpsAlertNotifyTypeSeparatesNodes(t *testing.T) { + alert := SmartOpsAlert{Key: "system_cpu", Node: "node-a", Status: smartOpsAlertStatusFiring} + nodeAType := smartOpsAlertNotifyType(alert) + alert.Node = "node-b" + nodeBType := smartOpsAlertNotifyType(alert) + + require.NotEqual(t, nodeAType, nodeBType) + require.Contains(t, nodeAType, "node-a") + require.Contains(t, nodeBType, "node-b") + + alert.Node = "node-a" + alert.Status = smartOpsAlertStatusResolved + require.NotEqual(t, nodeAType, smartOpsAlertNotifyType(alert), "recovery events must not consume the firing-event budget") +} + +func TestDeliverSmartOpsAlertRetriesOnlyFailedRecipients(t *testing.T) { + originalDelays := smartOpsAlertRetryDelays + originalLoader := smartOpsAlertLoadRecipients + originalLimit := smartOpsAlertCheckNotificationLimit + originalSender := smartOpsAlertSendToRecipient + t.Cleanup(func() { + smartOpsAlertRetryDelays = originalDelays + smartOpsAlertLoadRecipients = originalLoader + smartOpsAlertCheckNotificationLimit = originalLimit + smartOpsAlertSendToRecipient = originalSender + }) + + smartOpsAlertRetryDelays = []time.Duration{0, 0} + smartOpsAlertLoadRecipients = func() ([]smartOpsAlertRecipient, error) { + return []smartOpsAlertRecipient{{ID: 1}, {ID: 2}}, nil + } + limitChecks := map[int]int{} + smartOpsAlertCheckNotificationLimit = func(userID int, _ string) (bool, error) { + limitChecks[userID]++ + return true, nil + } + sendAttempts := map[int]int{} + var deliveredNotification dto.Notify + smartOpsAlertSendToRecipient = func(recipient smartOpsAlertRecipient, notification dto.Notify) error { + sendAttempts[recipient.ID]++ + if recipient.ID == 2 && sendAttempts[recipient.ID] < 3 { + return fmt.Errorf("temporary delivery failure") + } + deliveredNotification = notification + return nil + } + + err := deliverSmartOpsAlertNotification(SmartOpsAlert{ + Key: "system_cpu", + Node: "node-a", + Status: smartOpsAlertStatusFiring, + ObservedAt: time.Unix(400, 0), + }) + + require.NoError(t, err) + require.Equal(t, map[int]int{1: 1, 2: 1}, limitChecks, "a retry must not consume a new logical notification slot") + require.Equal(t, 1, sendAttempts[1], "successful administrators must not receive duplicates") + require.Equal(t, 3, sendAttempts[2], "only the failed administrator should be retried") + require.Contains(t, deliveredNotification.Content, "查看:/smart-ops/alerts") +} + +func TestDeliverSmartOpsAlertRetriesRecipientLookup(t *testing.T) { + originalDelays := smartOpsAlertRetryDelays + originalLoader := smartOpsAlertLoadRecipients + originalLimit := smartOpsAlertCheckNotificationLimit + originalSender := smartOpsAlertSendToRecipient + t.Cleanup(func() { + smartOpsAlertRetryDelays = originalDelays + smartOpsAlertLoadRecipients = originalLoader + smartOpsAlertCheckNotificationLimit = originalLimit + smartOpsAlertSendToRecipient = originalSender + }) + + smartOpsAlertRetryDelays = []time.Duration{0} + loadAttempts := 0 + smartOpsAlertLoadRecipients = func() ([]smartOpsAlertRecipient, error) { + loadAttempts++ + if loadAttempts == 1 { + return nil, fmt.Errorf("temporary database failure") + } + return []smartOpsAlertRecipient{{ID: 1}}, nil + } + smartOpsAlertCheckNotificationLimit = func(_ int, _ string) (bool, error) { return true, nil } + sent := 0 + smartOpsAlertSendToRecipient = func(_ smartOpsAlertRecipient, _ dto.Notify) error { + sent++ + return nil + } + + err := deliverSmartOpsAlertNotification(SmartOpsAlert{Key: "system_cpu", Node: "node-a"}) + + require.NoError(t, err) + require.Equal(t, 2, loadAttempts) + require.Equal(t, 1, sent) +} + +func TestSmartOpsAlertNotificationPoolRunsIndependentIncidentsConcurrentlyAndPreservesOrder(t *testing.T) { + firstStarted := make(chan struct{}) + releaseFirst := make(chan struct{}) + independentDelivered := make(chan struct{}, 1) + incidentOrder := make(chan string, 2) + + pool := newSmartOpsAlertNotificationPool(2, 4, func(alert SmartOpsAlert) error { + if alert.Key == "system_cpu" && alert.Node == "node-a" { + if alert.Status == smartOpsAlertStatusFiring { + close(firstStarted) + <-releaseFirst + } + incidentOrder <- alert.Status + return nil + } + independentDelivered <- struct{}{} + return nil + }) + t.Cleanup(pool.close) + + firing := SmartOpsAlert{Key: "system_cpu", Node: "node-a", Status: smartOpsAlertStatusFiring} + resolved := firing + resolved.Status = smartOpsAlertStatusResolved + require.Equal(t, pool.workerIndex(firing), pool.workerIndex(resolved), "one incident must stay on one ordered worker") + + independent := SmartOpsAlert{Key: "system_memory", Node: "node-b", Status: smartOpsAlertStatusFiring} + if pool.workerIndex(independent) == pool.workerIndex(firing) { + independent = SmartOpsAlert{Key: "system_disk", Node: "node-c", Status: smartOpsAlertStatusFiring} + } + require.NotEqual(t, pool.workerIndex(firing), pool.workerIndex(independent), "the regression fixture must use another worker") + + require.True(t, pool.enqueue(firing)) + select { + case <-firstStarted: + case <-time.After(time.Second): + t.Fatal("the first incident did not start delivery") + } + + require.True(t, pool.enqueue(resolved)) + require.True(t, pool.enqueue(independent)) + select { + case <-independentDelivered: + case <-time.After(time.Second): + t.Fatal("an unrelated alert was blocked behind a slow incident") + } + + close(releaseFirst) + for _, expected := range []string{smartOpsAlertStatusFiring, smartOpsAlertStatusResolved} { + select { + case actual := <-incidentOrder: + require.Equal(t, expected, actual) + case <-time.After(time.Second): + t.Fatalf("timed out waiting for %s delivery", expected) + } + } +} diff --git a/service/user_notify.go b/service/user_notify.go index 2f9d700c..5ca6d1c2 100644 --- a/service/user_notify.go +++ b/service/user_notify.go @@ -63,6 +63,17 @@ func NotifyUser(userId int, userEmail string, userSetting dto.UserSetting, data if !canSend { return fmt.Errorf("notification limit exceeded for user %d with type %s", userId, notifyType) } + return sendUserNotification(userId, userEmail, userSetting, data) +} + +// sendUserNotification performs one delivery after the caller has obtained a +// notification-limit permit. Keeping delivery separate lets SmartOps retry a +// transient transport failure without consuming another logical alert slot. +func sendUserNotification(userId int, userEmail string, userSetting dto.UserSetting, data dto.Notify) error { + notifyType := userSetting.NotifyType + if notifyType == "" { + notifyType = dto.NotifyTypeEmail + } switch notifyType { case dto.NotifyTypeEmail: diff --git a/setting/console_setting/validation.go b/setting/console_setting/validation.go index d6e4342c..7f96f791 100644 --- a/setting/console_setting/validation.go +++ b/setting/console_setting/validation.go @@ -1,13 +1,15 @@ package console_setting import ( - "encoding/json" "fmt" "net/url" "regexp" "sort" "strings" "time" + "unicode/utf16" + + "github.com/MAX-API-Next/MAX-API/common" ) var ( @@ -24,12 +26,16 @@ var ( func parseJSONArray(jsonStr string, typeName string) ([]map[string]interface{}, error) { var list []map[string]interface{} - if err := json.Unmarshal([]byte(jsonStr), &list); err != nil { + if err := common.UnmarshalJsonStr(jsonStr, &list); err != nil { return nil, fmt.Errorf("%s格式错误:%s", typeName, err.Error()) } return list, nil } +func exceedsMaxCharacters(s string, maxCharacters int) bool { + return len(utf16.Encode([]rune(s))) > maxCharacters +} + func validateURL(urlStr string, index int, itemType string) error { if !urlRegex.MatchString(urlStr) { return fmt.Errorf("第%d个%s的URL格式不正确", index, itemType) @@ -55,7 +61,7 @@ func getJSONList(jsonStr string) []map[string]interface{} { return []map[string]interface{}{} } var list []map[string]interface{} - json.Unmarshal([]byte(jsonStr), &list) + _ = common.UnmarshalJsonStr(jsonStr, &list) return list } @@ -110,13 +116,13 @@ func validateApiInfo(apiInfoStr string) error { return err } - if len(urlStr) > 500 { + if exceedsMaxCharacters(urlStr, 500) { return fmt.Errorf("第%d个API信息的URL长度不能超过500字符", i+1) } - if len(route) > 100 { + if exceedsMaxCharacters(route, 100) { return fmt.Errorf("第%d个API信息的线路描述长度不能超过100字符", i+1) } - if len(description) > 200 { + if exceedsMaxCharacters(description, 200) { return fmt.Errorf("第%d个API信息的说明长度不能超过200字符", i+1) } @@ -172,11 +178,11 @@ func validateAnnouncements(announcementsStr string) error { } } } - if len(content) > 500 { + if exceedsMaxCharacters(content, 500) { return fmt.Errorf("第%d个公告的内容长度不能超过500字符", i+1) } if extra, exists := ann["extra"]; exists { - if extraStr, ok := extra.(string); ok && len(extraStr) > 200 { + if extraStr, ok := extra.(string); ok && exceedsMaxCharacters(extraStr, 200) { return fmt.Errorf("第%d个公告的说明长度不能超过200字符", i+1) } } @@ -201,10 +207,10 @@ func validateFAQ(faqStr string) error { if !ok || answer == "" { return fmt.Errorf("第%d个FAQ缺少答案字段", i+1) } - if len(question) > 200 { + if exceedsMaxCharacters(question, 200) { return fmt.Errorf("第%d个FAQ的问题长度不能超过200字符", i+1) } - if len(answer) > 1000 { + if exceedsMaxCharacters(answer, 1000) { return fmt.Errorf("第%d个FAQ的答案长度不能超过1000字符", i+1) } } @@ -272,16 +278,16 @@ func validateUptimeKumaGroups(groupsStr string) error { return err } - if len(categoryName) > 50 { + if exceedsMaxCharacters(categoryName, 50) { return fmt.Errorf("第%d个分组的分类名称长度不能超过50字符", i+1) } - if len(urlStr) > 500 { + if exceedsMaxCharacters(urlStr, 500) { return fmt.Errorf("第%d个分组的URL长度不能超过500字符", i+1) } - if len(slug) > 100 { + if exceedsMaxCharacters(slug, 100) { return fmt.Errorf("第%d个分组的Slug长度不能超过100字符", i+1) } - if len(description) > 200 { + if exceedsMaxCharacters(description, 200) { return fmt.Errorf("第%d个分组的描述长度不能超过200字符", i+1) } diff --git a/setting/console_setting/validation_test.go b/setting/console_setting/validation_test.go new file mode 100644 index 00000000..a0739938 --- /dev/null +++ b/setting/console_setting/validation_test.go @@ -0,0 +1,29 @@ +package console_setting + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestExceedsMaxCharactersMatchesJavaScriptUTF16Length(t *testing.T) { + require.False(t, exceedsMaxCharacters("abc", 3)) + require.False(t, exceedsMaxCharacters("中文", 2)) + require.False(t, exceedsMaxCharacters("é", 1)) + require.True(t, exceedsMaxCharacters("😀", 1)) + require.False(t, exceedsMaxCharacters("😀", 2)) +} + +func TestValidateFAQUsesUTF16CodeUnits(t *testing.T) { + valid := `[{"question":"` + strings.Repeat("中", 200) + `","answer":"ok"}]` + require.NoError(t, validateFAQ(valid)) + + tooLongEmoji := `[{"question":"` + strings.Repeat("😀", 101) + `","answer":"ok"}]` + require.ErrorContains(t, validateFAQ(tooLongEmoji), "不能超过200字符") +} + +func TestParseJSONArrayReturnsInvalidJSONError(t *testing.T) { + _, err := parseJSONArray(`[{`, "FAQ信息") + require.ErrorContains(t, err, "FAQ信息格式错误") +} diff --git a/tools/jsonwrapcheck/allowlist.txt b/tools/jsonwrapcheck/allowlist.txt index 7b58707e..6e1cf982 100644 --- a/tools/jsonwrapcheck/allowlist.txt +++ b/tools/jsonwrapcheck/allowlist.txt @@ -87,10 +87,6 @@ relay/channel/minimax/adaptor.go|*Adaptor.ConvertAudioRequest|Marshal|72d25474ab relay/channel/minimax/adaptor_test.go|TestConvertImageRequest|Marshal|f9f22be55e3cc256321948a6831e6d8f59f28dca147ab5d96d122bb05bb77b67 relay/channel/minimax/adaptor_test.go|TestConvertImageRequest|Unmarshal|fbd89e33afec938e40c3b365fc286bb279c1440145c68098c54ab715bd85b119 relay/channel/mokaai/relay-mokaai.go|mokaEmbeddingHandler|Unmarshal|6c88b37cda6b69cb880852f57074d792626b82bd771acbab3a46754f80631684 -relay/channel/ollama/relay-ollama.go|openAIChatToOllamaChat|Unmarshal|cdc0ef0788af1c25be7a997463473148c1a5660910f5d793f0405b3ce78d3343 -relay/channel/ollama/relay-ollama.go|openAIChatToOllamaChat|Unmarshal|07a078b20e64cb7aab1e475cef9ea2651d4a2c279ae38cdec64dd0bf01ea3083 -relay/channel/ollama/relay-ollama.go|openAIToGenerate|Unmarshal|cdc0ef0788af1c25be7a997463473148c1a5660910f5d793f0405b3ce78d3343 -relay/channel/ollama/relay-ollama.go|FetchOllamaVersion|Unmarshal|63a9dd239b9dc0824369644993015b931b45ec167ed8555e7d530ce84d5c8678 relay/channel/openai/adaptor.go|*Adaptor.ConvertOpenAIRequest|Unmarshal|44a1603ee0e1a8e1df85d054b5b756d4357abc38dbe367a0fb9192cab91c1e25 relay/channel/palm/relay-palm.go|palmStreamHandler|Unmarshal|9e6a91a6205bc2652fded36139425ad52ae7c49ef94ccd023817fca36089ced0 relay/channel/palm/relay-palm.go|palmStreamHandler|Marshal|92981973316a59184c1f9d4fdca6fe3a1045b5da6a30f89abb52b516f1849571 @@ -117,8 +113,6 @@ service/passkey/session.go|PopSessionData|Unmarshal|f2cd8d698a8b3baee9426fd333d1 service/user_notify.go|sendGotifyNotify|Marshal|2af017bb4f8b2b336f7069e3a6f219c0fb04cddbcb36aea984cee19b0af05e40 setting/chat.go|UpdateChatsByJsonString|Unmarshal|9b652f37646cc5b0c6883ad42cfa81aa57bf3ae9ed3f64bf59c0ca4e9024983e setting/chat.go|Chats2JsonString|Marshal|6d15cc12f47b577c984fd31bbc9ca3e9bbb3702d28b79ac76d58e9aa32e79fdc -setting/console_setting/validation.go|parseJSONArray|Unmarshal|403028caafc69490779d75b66909a63579743ae6b77481ab45fba2b65aa34805 -setting/console_setting/validation.go|getJSONList|Unmarshal|403028caafc69490779d75b66909a63579743ae6b77481ab45fba2b65aa34805 setting/rate_limit.go|ModelRequestRateLimitGroup2JSONString|Marshal|c72783cb7874ede43c0e880006216b4e7a02605d568d41c3360f5da2674d41dc setting/rate_limit.go|UpdateModelRequestRateLimitGroupByJSONString|Unmarshal|7407dc7c4c3c97a8dba391911f2c435cfc531476870e4deef40b633f4cf49463 setting/rate_limit.go|CheckModelRequestRateLimitGroup|Unmarshal|e3bd25b5c46d357559a707ed7efbf4d2659495da494208dce96b6de39f6295d1 diff --git a/web/default/package.json b/web/default/package.json index dd867ab4..e768ef99 100644 --- a/web/default/package.json +++ b/web/default/package.json @@ -7,6 +7,7 @@ "dev": "rsbuild dev", "build": "rsbuild build && node scripts/verify-tailwind-build.mjs", "build:check": "tsc -b && rsbuild build && node scripts/verify-tailwind-build.mjs", + "test": "bun scripts/run-tests.mjs", "typecheck": "tsc -b", "lint": "eslint .", "preview": "rsbuild preview", diff --git a/web/default/scripts/run-tests.mjs b/web/default/scripts/run-tests.mjs new file mode 100644 index 00000000..e2cbdb71 --- /dev/null +++ b/web/default/scripts/run-tests.mjs @@ -0,0 +1,329 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import path from 'node:path' +import { readdir } from 'node:fs/promises' +import { fileURLToPath } from 'node:url' + +const scriptDirectory = path.dirname(fileURLToPath(import.meta.url)) +const projectRoot = path.resolve(scriptDirectory, '..') +const testGlob = new Bun.Glob( + '**/*{.test,_test,.spec,_spec}.{ts,tsx,js,jsx,mjs,cjs,mts,cts}' +) +const ignoredDirectories = new Set([ + '.git', + 'coverage', + 'dist', + 'node_modules', +]) +const flagsWithSeparateValues = new Set([ + '-t', + '--conditions', + '--coverage-dir', + '--coverage-reporter', + '--define', + '--env-file', + '--loader', + '--max-concurrency', + '--parallel-delay', + '--path-ignore-patterns', + '--preload', + '--reporter', + '--reporter-outfile', + '--rerun-each', + '--retry', + '--seed', + '--test-name-pattern', + '--timeout', + '--tsconfig-override', +]) +const optionalValueFlags = new Map([ + ['--bail', (value) => /^\d+$/.test(value)], + ['--changed', () => true], + ['--parallel', (value) => /^\d+$/.test(value)], +]) +const textDecoder = new TextDecoder() +const shardValuePattern = /^(\d+)\/(\d+)$/ +const maxConcurrencyValuePattern = /^\d+$/ +const globalPropertyPattern = + String.raw`globalThis(?:\s*\.\s*[A-Za-z_$][\w$]*|\s*\[[^\]]*\])+` +const globalMutationPattern = new RegExp( + [ + String.raw`(?:Object\.(?:assign|defineProperties|defineProperty)|Reflect\.(?:defineProperty|deleteProperty|set))\s*\(\s*globalThis\b`, + String.raw`delete\b\s*(?:\(\s*)?globalThis\b`, + String.raw`${globalPropertyPattern}\s*(?:\*\*=|>>>=|<<=|>>=|&&=|\|\|=|\?\?=|[+\-*/%&^|]=|=(?!=))`, + String.raw`${globalPropertyPattern}[^\S\r\n]*(?:\+\+|--)`, + String.raw`(?:\+\+|--)\s*${globalPropertyPattern}`, + ].join('|') +) + +function normalizePath(file) { + return file.replaceAll('\\', '/') +} + +async function collectTestFiles( + root, + relativeDirectory, + readDirectory, + testFiles +) { + const directory = path.join(root, relativeDirectory) + const entries = await readDirectory(directory, { withFileTypes: true }) + + for (const entry of entries) { + const relativePath = relativeDirectory + ? `${normalizePath(relativeDirectory)}/${entry.name}` + : entry.name + if (entry.isDirectory()) { + if (!ignoredDirectories.has(entry.name)) { + await collectTestFiles(root, relativePath, readDirectory, testFiles) + } + continue + } + if (entry.isFile() && testGlob.match(relativePath)) { + testFiles.push(relativePath) + } + } +} + +export async function discoverTestFiles(root = projectRoot, readDirectory = readdir) { + const testFiles = [] + await collectTestFiles(root, '', readDirectory, testFiles) + return [...new Set(testFiles)].sort() +} + +export function splitTestArguments(args) { + const bunArguments = [] + const fileFilters = [] + let positionalOnly = false + + for (let index = 0; index < args.length; index += 1) { + const argument = args[index] + if (!positionalOnly && argument === '--') { + positionalOnly = true + continue + } + if (!positionalOnly && argument.startsWith('-')) { + bunArguments.push(argument) + if ( + argument === '--shard' && + index + 1 < args.length && + shardValuePattern.test(args[index + 1]) + ) { + bunArguments.push(args[index + 1]) + index += 1 + } else if ( + flagsWithSeparateValues.has(argument) && + index + 1 < args.length + ) { + bunArguments.push(args[index + 1]) + index += 1 + } else if ( + optionalValueFlags.has(argument) && + index + 1 < args.length && + !args[index + 1].startsWith('-') && + optionalValueFlags.get(argument)(args[index + 1]) + ) { + bunArguments.push(args[index + 1]) + index += 1 + } + continue + } + fileFilters.push(argument) + } + + return { bunArguments, fileFilters } +} + +function matchesFileFilter(file, filter, root) { + const normalizedFile = normalizePath(file) + const normalizedFilter = normalizePath(filter).replace(/^\.\//, '') + const relativeFilter = path.isAbsolute(filter) + ? normalizePath(path.relative(root, filter)) + : normalizedFilter + + if (/[*?[\]{}]/.test(relativeFilter)) { + return new Bun.Glob(relativeFilter).match(normalizedFile) + } + return normalizedFile.includes(relativeFilter) +} + +export function selectTestFiles(files, filters, root = projectRoot) { + if (filters.length === 0) return files + return files.filter((file) => + filters.some((filter) => matchesFileFilter(file, filter, root)) + ) +} + +function parseShardValue(value) { + const match = shardValuePattern.exec(value) + if (!match) throw new Error(`Invalid shard value: ${value}`) + + const index = Number(match[1]) + const total = Number(match[2]) + if (index < 1 || total < 1 || index > total) { + throw new Error(`Invalid shard value: ${value}`) + } + return { index, total } +} + +export function extractShardArgument(args) { + const bunArguments = [] + let shard + + for (let index = 0; index < args.length; index += 1) { + const argument = args[index] + if (argument === '--shard') { + if (index + 1 >= args.length) throw new Error('--shard requires a value') + shard = parseShardValue(args[index + 1]) + index += 1 + continue + } + if (argument.startsWith('--shard=')) { + shard = parseShardValue(argument.slice('--shard='.length)) + continue + } + bunArguments.push(argument) + } + + return { bunArguments, shard } +} + +export function applyTestShard(files, shard) { + if (!shard) return files + return files.filter((_, index) => index % shard.total === shard.index - 1) +} + +export function usesIsolatedEnvironment(file, source) { + return ( + file.endsWith('.tsx') || + source.includes('createReactTestEnvironment') || + source.includes("from 'jsdom'") || + source.includes('from "jsdom"') || + globalMutationPattern.test(source) + ) +} + +async function partitionTestFiles(files) { + const sharedFiles = [] + const isolatedFiles = [] + + for (const file of files) { + const source = await Bun.file(path.join(projectRoot, file)).text() + if (usesIsolatedEnvironment(file, source)) isolatedFiles.push(file) + else sharedFiles.push(file) + } + return { sharedFiles, isolatedFiles } +} + +function writeChildOutput(result) { + if (result.stdout.length > 0) process.stdout.write(result.stdout) + if (result.stderr.length > 0) process.stderr.write(result.stderr) +} + +function validateMaxConcurrencyValue(value) { + if (value === undefined) { + throw new Error('--max-concurrency requires a numeric value') + } + if (!maxConcurrencyValuePattern.test(value)) { + throw new Error(`Invalid --max-concurrency value: ${value}`) + } +} + +export function buildTestCommand(files, bunArguments) { + const forwardedArguments = [] + for (let index = 0; index < bunArguments.length; index += 1) { + const argument = bunArguments[index] + if (argument === '--max-concurrency') { + validateMaxConcurrencyValue(bunArguments[index + 1]) + index += 1 + continue + } + if (argument.startsWith('--max-concurrency=')) { + validateMaxConcurrencyValue(argument.slice('--max-concurrency='.length)) + continue + } + forwardedArguments.push(argument) + } + + return [ + process.execPath, + 'test', + ...forwardedArguments, + '--max-concurrency=1', + ...files, + ] +} + +function runTestFiles(files, label, bunArguments) { + if (files.length === 0) return 'skipped' + + console.log(`\n==> ${label} (${files.length} file${files.length === 1 ? '' : 's'})`) + const result = Bun.spawnSync({ + cmd: buildTestCommand(files, bunArguments), + cwd: projectRoot, + env: process.env, + stdout: 'pipe', + stderr: 'pipe', + }) + writeChildOutput(result) + if (result.exitCode === 0) return 'passed' + + const output = `${textDecoder.decode(result.stdout)}\n${textDecoder.decode(result.stderr)}` + if (output.includes('matched 0 tests')) return 'no-tests' + process.exit(result.exitCode) +} + +async function main() { + const parsedArguments = splitTestArguments(process.argv.slice(2)) + const shardArguments = extractShardArgument(parsedArguments.bunArguments) + const bunArguments = shardArguments.bunArguments + const discoveredFiles = await discoverTestFiles() + const filteredFiles = selectTestFiles(discoveredFiles, parsedArguments.fileFilters) + const selectedFiles = applyTestShard(filteredFiles, shardArguments.shard) + const passWithNoTests = bunArguments.includes('--pass-with-no-tests') + + if (shardArguments.shard) { + console.log( + `--shard=${shardArguments.shard.index}/${shardArguments.shard.total}: running ${selectedFiles.length}/${filteredFiles.length} test files` + ) + } + + if (selectedFiles.length === 0) { + console.error('No test files matched the supplied filters.') + process.exit(passWithNoTests ? 0 : 1) + } + + const { sharedFiles, isolatedFiles } = await partitionTestFiles(selectedFiles) + let passedRunCount = 0 + if (runTestFiles(sharedFiles, 'Shared-state-safe tests', bunArguments) === 'passed') { + passedRunCount += 1 + } + for (const file of isolatedFiles) { + if ( + runTestFiles([file], `Isolated browser test: ${file}`, bunArguments) === + 'passed' + ) { + passedRunCount += 1 + } + } + + if (passedRunCount === 0 && !passWithNoTests) process.exit(1) +} + +if (import.meta.main) await main() diff --git a/web/default/scripts/run-tests.test.mjs b/web/default/scripts/run-tests.test.mjs new file mode 100644 index 00000000..c3f39ab7 --- /dev/null +++ b/web/default/scripts/run-tests.test.mjs @@ -0,0 +1,314 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import assert from 'node:assert/strict' +import { mkdtemp, mkdir, readdir, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import path from 'node:path' +import { test } from 'node:test' +import { + applyTestShard, + buildTestCommand, + discoverTestFiles, + extractShardArgument, + selectTestFiles, + splitTestArguments, + usesIsolatedEnvironment, +} from './run-tests.mjs' + +test('discovers every supported test extension outside only generated directories', async () => { + const root = await mkdtemp(path.join(tmpdir(), 'max-api-run-tests-')) + try { + const files = [ + 'scripts/runner.test.mjs', + 'scripts/legacy_test.cts', + 'src/component.spec.jsx', + 'src/legacy_spec.mts', + 'tests/example.test.js', + '.git/hooks/internal.test.js', + 'coverage/report.test.js', + 'node_modules/dependency.test.js', + 'dist/generated.spec.mjs', + ] + for (const file of files) { + const absolutePath = path.join(root, file) + await mkdir(path.dirname(absolutePath), { recursive: true }) + await writeFile(absolutePath, '') + } + + const visitedDirectories = [] + const trackingReadDirectory = async (...args) => { + const relativeDirectory = path.relative(root, args[0]).replaceAll('\\', '/') + visitedDirectories.push(relativeDirectory || '.') + return readdir(...args) + } + + assert.deepEqual(await discoverTestFiles(root, trackingReadDirectory), [ + 'scripts/legacy_test.cts', + 'scripts/runner.test.mjs', + 'src/component.spec.jsx', + 'src/legacy_spec.mts', + 'tests/example.test.js', + ]) + assert.deepEqual(visitedDirectories.sort(), ['.', 'scripts', 'src', 'tests']) + } finally { + await rm(root, { force: true, recursive: true }) + } +}) + +test('separates Bun flags from file filters without dropping flag values', () => { + assert.deepEqual( + splitTestArguments([ + '--test-name-pattern', + 'renders details', + '--coverage', + '--parallel', + '1', + '--preload', + './test/setup.ts', + '--env-file', + '.env.test', + 'src/features/example.test.ts', + ]), + { + bunArguments: [ + '--test-name-pattern', + 'renders details', + '--coverage', + '--parallel', + '1', + '--preload', + './test/setup.ts', + '--env-file', + '.env.test', + ], + fileFilters: ['src/features/example.test.ts'], + } + ) + + assert.deepEqual(splitTestArguments(['--bail', 'src/example.test.ts']), { + bunArguments: ['--bail'], + fileFilters: ['src/example.test.ts'], + }) + assert.deepEqual( + splitTestArguments(['--shard', '2/3', 'src/example.test.ts']), + { + bunArguments: ['--shard', '2/3'], + fileFilters: ['src/example.test.ts'], + } + ) + assert.deepEqual(splitTestArguments(['--changed', 'origin/main']), { + bunArguments: ['--changed', 'origin/main'], + fileFilters: [], + }) + assert.deepEqual( + splitTestArguments([ + '--conditions', + 'development', + '--define', + 'MAX_API_TEST_VALUE=1', + '--loader', + '.txt:text', + '--tsconfig-override', + './tsconfig.test.json', + 'src/example.test.ts', + ]), + { + bunArguments: [ + '--conditions', + 'development', + '--define', + 'MAX_API_TEST_VALUE=1', + '--loader', + '.txt:text', + '--tsconfig-override', + './tsconfig.test.json', + ], + fileFilters: ['src/example.test.ts'], + } + ) +}) + +test('applies plain and glob file filters to the discovered test list', () => { + const files = [ + 'scripts/run-tests.test.mjs', + 'src/features/a/alpha.test.ts', + 'src/features/b/beta.spec.tsx', + ] + + assert.deepEqual(selectTestFiles(files, ['alpha']), [ + 'src/features/a/alpha.test.ts', + ]) + assert.deepEqual(selectTestFiles(files, ['scripts/*.test.mjs']), [ + 'scripts/run-tests.test.mjs', + ]) +}) + +test('applies Bun shards across the complete selected file list', () => { + const parsed = extractShardArgument(['--coverage', '--shard=2/3']) + assert.deepEqual(parsed, { + bunArguments: ['--coverage'], + shard: { index: 2, total: 3 }, + }) + assert.deepEqual( + applyTestShard(['a.test.ts', 'b.test.ts', 'c.test.ts', 'd.test.ts'], parsed.shard), + ['b.test.ts'] + ) +}) + +test('keeps the child test process serial when callers set concurrency', () => { + const files = ['src/example.test.ts'] + const command = buildTestCommand(files, [ + '--coverage', + '--max-concurrency', + '8', + '--timeout', + '5000', + '--max-concurrency=4', + ]) + + assert.deepEqual(command, [ + process.execPath, + 'test', + '--coverage', + '--timeout', + '5000', + '--max-concurrency=1', + ...files, + ]) + assert.equal( + command.filter((argument) => argument.startsWith('--max-concurrency')).length, + 1 + ) + assert.ok(command.indexOf('--max-concurrency=1') < command.indexOf(files[0])) +}) + +test('rejects malformed caller concurrency values', () => { + const files = ['src/example.test.ts'] + + assert.throws( + () => buildTestCommand(files, ['--max-concurrency']), + /--max-concurrency requires a numeric value/ + ) + assert.throws( + () => buildTestCommand(files, ['--max-concurrency', 'many']), + /Invalid --max-concurrency value: many/ + ) + assert.throws( + () => buildTestCommand(files, ['--max-concurrency=']), + /Invalid --max-concurrency value:/ + ) + assert.throws( + () => buildTestCommand(files, ['--max-concurrency=many']), + /Invalid --max-concurrency value: many/ + ) +}) + +test('isolates tests that mutate browser globals even when they are plain TypeScript', () => { + assert.equal( + usesIsolatedEnvironment( + 'src/example.test.ts', + "Object.defineProperty(\n globalThis,\n 'window',\n { value: {} }\n)" + ), + true + ) + for (const source of [ + "Reflect.set(globalThis, 'TEST_FLAG', '1')", + "Object.assign(globalThis, { TEST_FLAG: '1' })", + "Object.defineProperties(globalThis, { TEST_FLAG: { value: '1' } })", + "Reflect.defineProperty(globalThis, 'TEST_FLAG', { value: '1' })", + 'globalThis.count += 1', + 'globalThis.count -= 1', + 'globalThis.count *= 1', + 'globalThis.count /= 1', + 'globalThis.count %= 1', + 'globalThis.count **= 1', + 'globalThis.count <<= 1', + 'globalThis.count >>= 1', + 'globalThis.count >>>= 1', + 'globalThis.count &= 1', + 'globalThis.count ^= 1', + 'globalThis.count |= 1', + 'globalThis.count &&= 1', + 'globalThis.count ||= 1', + "globalThis['count'] ??= 1", + 'globalThis.count++', + '++globalThis.count', + 'globalThis.count--', + '--globalThis.count', + "globalThis.process.env.TEST_FLAG = '1'", + "globalThis['process'].env['TEST_FLAG'] ||= '1'", + "globalThis.process['env'].TEST_FLAG++", + ]) { + assert.equal(usesIsolatedEnvironment('src/example.test.ts', source), true) + } + assert.equal( + usesIsolatedEnvironment('src/example.test.ts', 'delete globalThis.window'), + true + ) + assert.equal( + usesIsolatedEnvironment( + 'src/example.test.ts', + "globalThis['window'] = {}" + ), + true + ) + assert.equal( + usesIsolatedEnvironment('src/example.test.ts', 'globalThis[propertyName] = {}'), + true + ) + assert.equal( + usesIsolatedEnvironment( + 'src/example.test.ts', + "globalThis[\n 'window'\n] = {}" + ), + true + ) + assert.equal( + usesIsolatedEnvironment( + 'src/example.test.ts', + "globalThis['window'] === existingWindow" + ), + false + ) + assert.equal( + usesIsolatedEnvironment( + 'src/example.test.ts', + 'globalThis.count\n++otherCount' + ), + false + ) + assert.equal( + usesIsolatedEnvironment( + 'src/example.test.ts', + "globalThis['window'] === existingWindow\nvalues[index] = nextValue" + ), + false + ) + assert.equal( + usesIsolatedEnvironment( + 'src/example.test.ts', + "globalThis.process.env.TEST_FLAG === '1'\nvalues[index] = nextValue" + ), + false + ) + assert.equal( + usesIsolatedEnvironment('src/example.test.ts', 'assert.equal(1, 1)'), + false + ) +}) diff --git a/web/default/src/components/layout/config/smart-ops.config.test.ts b/web/default/src/components/layout/config/smart-ops.config.test.ts new file mode 100644 index 00000000..cff8e0f4 --- /dev/null +++ b/web/default/src/components/layout/config/smart-ops.config.test.ts @@ -0,0 +1,50 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import type { TFunction } from 'i18next' +import assert from 'node:assert/strict' +import { describe, test } from 'node:test' +import { ROLE } from '@/lib/roles' +import { buildSmartOpsNavGroups } from './smart-ops.config' + +const translate = ((key: string) => key) as TFunction + +describe('Smart Operations navigation', () => { + test('keeps channel and model performance available to administrators', () => { + const [group] = buildSmartOpsNavGroups(translate, ROLE.ADMIN) + + assert.deepEqual( + group.items.map((item) => item.title), + ['Active Alerts', 'Channel performance', 'Model performance'] + ) + }) + + test('adds system information only for the super administrator', () => { + const [group] = buildSmartOpsNavGroups(translate, ROLE.SUPER_ADMIN) + + assert.deepEqual( + group.items.map((item) => item.title), + [ + 'Active Alerts', + 'Channel performance', + 'Model performance', + 'System Info', + ] + ) + }) +}) diff --git a/web/default/src/components/layout/config/smart-ops.config.ts b/web/default/src/components/layout/config/smart-ops.config.ts new file mode 100644 index 00000000..d4213662 --- /dev/null +++ b/web/default/src/components/layout/config/smart-ops.config.ts @@ -0,0 +1,78 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import { type TFunction } from 'i18next' +import { Bot, Gauge, ServerCog, TriangleAlert } from 'lucide-react' +import { useAuthStore } from '@/stores/auth-store' +import { ROLE } from '@/lib/roles' +import type { NavGroup, SidebarView } from '../types' + +export function buildSmartOpsNavGroups( + t: TFunction, + userRole: number | undefined +): NavGroup[] { + const isSuperAdmin = userRole === ROLE.SUPER_ADMIN + const items = [ + { + title: t('Active Alerts'), + url: '/smart-ops/alerts' as const, + icon: TriangleAlert, + }, + { + title: t('Channel performance'), + url: '/smart-ops/channel-performance' as const, + icon: Gauge, + }, + { + title: t('Model performance'), + url: '/smart-ops/model-performance' as const, + icon: Bot, + }, + ...(isSuperAdmin + ? [ + { + title: t('System Info'), + url: '/smart-ops/system-info' as const, + icon: ServerCog, + }, + ] + : []), + ] + + return [ + { + id: 'smart-operations', + title: t('Smart Operations Center'), + items, + }, + ] +} + +function getSmartOpsNavGroups(t: TFunction): NavGroup[] { + return buildSmartOpsNavGroups(t, useAuthStore.getState().auth.user?.role) +} + +export const SMART_OPS_VIEW: SidebarView = { + id: 'smart-ops', + pathPattern: /^\/smart-ops(\/|$)/, + parent: { + to: '/dashboard/overview', + label: 'Back to Dashboard', + }, + getNavGroups: getSmartOpsNavGroups, +} diff --git a/web/default/src/components/layout/lib/sidebar-view-registry.ts b/web/default/src/components/layout/lib/sidebar-view-registry.ts index fde22838..54280011 100644 --- a/web/default/src/components/layout/lib/sidebar-view-registry.ts +++ b/web/default/src/components/layout/lib/sidebar-view-registry.ts @@ -17,6 +17,7 @@ along with this program. If not, see . For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues */ import { type TFunction } from 'i18next' +import { SMART_OPS_VIEW } from '../config/smart-ops.config' import { SYSTEM_SETTINGS_VIEW } from '../config/system-settings.config' import type { NavGroup, SidebarView } from '../types' @@ -29,7 +30,10 @@ import type { NavGroup, SidebarView } from '../types' * * Match priority is array order; the first matching `pathPattern` wins. */ -const SIDEBAR_VIEWS: readonly SidebarView[] = [SYSTEM_SETTINGS_VIEW] +const SIDEBAR_VIEWS: readonly SidebarView[] = [ + SMART_OPS_VIEW, + SYSTEM_SETTINGS_VIEW, +] /** * Resolve the active nested view for the given path. diff --git a/web/default/src/components/profile-dropdown.tsx b/web/default/src/components/profile-dropdown.tsx index 5b405f8a..0749121a 100644 --- a/web/default/src/components/profile-dropdown.tsx +++ b/web/default/src/components/profile-dropdown.tsx @@ -47,7 +47,9 @@ export function ProfileDropdown() { const { displayName, roleLabel } = useUserDisplay(user) const isSuperAdmin = user?.role === ROLE.SUPER_ADMIN const isWalletVisible = useIsSidebarModuleVisible('/wallet') - const isSystemInfoVisible = useIsSidebarModuleVisible('/system-info') + const isSystemInfoVisible = useIsSidebarModuleVisible( + '/smart-ops/system-info' + ) const avatarName = user?.username || displayName const avatarFallback = getUserAvatarFallback(avatarName) const avatarFallbackStyle = useMemo( @@ -116,7 +118,7 @@ export function ProfileDropdown() { {isSuperAdmin && isSystemInfoVisible && ( navigate({ to: '/system-info' })} + onClick={() => navigate({ to: '/smart-ops/system-info' })} > {t('System Info')} diff --git a/web/default/src/features/auth/lib/turnstile-request.test.ts b/web/default/src/features/auth/lib/turnstile-request.test.ts index 50e7dc5e..50386644 100644 --- a/web/default/src/features/auth/lib/turnstile-request.test.ts +++ b/web/default/src/features/auth/lib/turnstile-request.test.ts @@ -19,6 +19,7 @@ For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API import assert from 'node:assert/strict' import { describe, test } from 'node:test' import { + consumeTurnstileToken, getTurnstileHeaders, TURNSTILE_TOKEN_HEADER, } from './turnstile-request' @@ -34,3 +35,17 @@ describe('getTurnstileHeaders', () => { assert.deepEqual(getTurnstileHeaders(''), {}) }) }) + +describe('consumeTurnstileToken', () => { + test('uses each token once and requires a refreshed token for retry', () => { + const first = consumeTurnstileToken('first-token', true) + assert.deepEqual(first, { + submittedToken: 'first-token', + nextToken: '', + shouldRefreshWidget: true, + }) + const second = consumeTurnstileToken('second-token', true) + assert.equal(second.submittedToken, 'second-token') + assert.equal(second.shouldRefreshWidget, true) + }) +}) diff --git a/web/default/src/features/auth/lib/turnstile-request.ts b/web/default/src/features/auth/lib/turnstile-request.ts index b8b5faf8..22e11dcf 100644 --- a/web/default/src/features/auth/lib/turnstile-request.ts +++ b/web/default/src/features/auth/lib/turnstile-request.ts @@ -23,3 +23,20 @@ export function getTurnstileHeaders(token?: string): Record { const value = token?.trim() return value ? { [TURNSTILE_TOKEN_HEADER]: value } : {} } + +export type ConsumeTurnstileTokenResult = { + submittedToken: string + nextToken: string + shouldRefreshWidget: boolean +} + +export function consumeTurnstileToken( + token: string, + enabled: boolean +): ConsumeTurnstileTokenResult { + return { + submittedToken: token, + nextToken: enabled ? '' : token, + shouldRefreshWidget: enabled, + } +} diff --git a/web/default/src/features/auth/sign-in/components/user-auth-form.tsx b/web/default/src/features/auth/sign-in/components/user-auth-form.tsx index 1b3981b2..8d21114f 100644 --- a/web/default/src/features/auth/sign-in/components/user-auth-form.tsx +++ b/web/default/src/features/auth/sign-in/components/user-auth-form.tsx @@ -58,6 +58,7 @@ import { OAuthProviders } from '@/features/auth/components/oauth-providers' import { loginFormSchema } from '@/features/auth/constants' import { useAuthRedirect } from '@/features/auth/hooks/use-auth-redirect' import { useTurnstile } from '@/features/auth/hooks/use-turnstile' +import { consumeTurnstileToken } from '@/features/auth/lib/turnstile-request' import { beginPasskeyLogin, finishPasskeyLogin } from '@/features/auth/passkey' import type { AuthFormProps } from '@/features/auth/types' @@ -75,6 +76,7 @@ export function UserAuthForm({ const [isPasskeyLoading, setIsPasskeyLoading] = useState(false) const [isWeChatDialogOpen, setIsWeChatDialogOpen] = useState(false) const [isWeChatSubmitting, setIsWeChatSubmitting] = useState(false) + const [turnstileWidgetKey, setTurnstileWidgetKey] = useState(0) const legalConsentErrorMessage = t('Please agree to the legal terms first') const loginFailedMessage = t('Login failed') @@ -171,13 +173,21 @@ export function UserAuthForm({ } if (!validateTurnstile()) return + const turnstileSubmission = consumeTurnstileToken( + turnstileToken, + isTurnstileEnabled + ) + if (turnstileSubmission.shouldRefreshWidget) { + setTurnstileToken(turnstileSubmission.nextToken) + setTurnstileWidgetKey((current) => current + 1) + } setIsLoading(true) try { const res = await login({ username: data.username, password: data.password, - turnstile: turnstileToken, + turnstile: turnstileSubmission.submittedToken, }) if (res.success) { @@ -421,8 +431,10 @@ export function UserAuthForm({ {isTurnstileEnabled && (
setTurnstileToken('')} />
)} diff --git a/web/default/src/features/channels/components/dialogs/fetch-models-dialog.tsx b/web/default/src/features/channels/components/dialogs/fetch-models-dialog.tsx index 010c50c6..cc6e883a 100644 --- a/web/default/src/features/channels/components/dialogs/fetch-models-dialog.tsx +++ b/web/default/src/features/channels/components/dialogs/fetch-models-dialog.tsx @@ -59,17 +59,27 @@ function normalizeModelNameList(models: readonly string[]): string[] { ) } -type FetchModelsDialogProps = { +type FetchModelsDialogBaseProps = { open: boolean onOpenChange: (open: boolean) => void - onModelsSelected?: (models: string[]) => void redirectModels?: string[] redirectSourceModels?: string[] customFetcher?: () => Promise - existingModelsOverride?: string[] channelName?: string | null } +type FetchModelsDialogProps = FetchModelsDialogBaseProps & + ( + | { + onModelsSelected: (models: string[]) => void + existingModelsOverride: string[] + } + | { + onModelsSelected?: undefined + existingModelsOverride?: undefined + } + ) + export function FetchModelsDialog({ open, onOpenChange, diff --git a/web/default/src/features/channels/components/drawers/channel-mutate-drawer.tsx b/web/default/src/features/channels/components/drawers/channel-mutate-drawer.tsx index 3d093319..7e3ca5a5 100644 --- a/web/default/src/features/channels/components/drawers/channel-mutate-drawer.tsx +++ b/web/default/src/features/channels/components/drawers/channel-mutate-drawer.tsx @@ -135,6 +135,7 @@ import { VIDEO_TASK_CHANNEL_TYPES, CHANNEL_TYPE_ADVANCED_CUSTOM, getAdvancedCustomStats, + fetchDialogExistingModels, } from '../../lib' import { collectInvalidStatusCodeEntries, @@ -480,6 +481,10 @@ export function ChannelMutateDrawer({ () => parseModelsString(currentModels), [currentModels] ) + const fetchDialogModelOverride = useMemo( + () => fetchDialogExistingModels(currentModelsArray), + [currentModelsArray] + ) const currentTypeLabel = useMemo( () => @@ -3941,11 +3946,7 @@ export function ChannelMutateDrawer({ redirectSourceModels={redirectModelKeyList} customFetcher={!isEditing ? createModeFetcher : undefined} channelName={!isEditing ? currentName?.trim() : undefined} - existingModelsOverride={ - !isEditing - ? parseModelsString(form.getValues('models') || '') - : undefined - } + existingModelsOverride={fetchDialogModelOverride} /> . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import assert from 'node:assert/strict' +import { describe, test } from 'node:test' +import { fetchDialogExistingModels } from './channel-editor-state' + +describe('channel fetch-model dialog state', () => { + test('uses the current unsaved form models in edit mode', () => { + const currentModels = ['saved-model', 'unsaved-model'] + const existingModels = fetchDialogExistingModels(currentModels) + + assert.deepEqual(existingModels, currentModels) + assert.notEqual(existingModels, currentModels) + }) +}) diff --git a/web/default/src/features/channels/lib/channel-editor-state.ts b/web/default/src/features/channels/lib/channel-editor-state.ts new file mode 100644 index 00000000..7c64c3b0 --- /dev/null +++ b/web/default/src/features/channels/lib/channel-editor-state.ts @@ -0,0 +1,24 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ + +export function fetchDialogExistingModels( + currentModels: readonly string[] +): string[] { + return [...currentModels] +} diff --git a/web/default/src/features/channels/lib/index.ts b/web/default/src/features/channels/lib/index.ts index 29381b29..a7fa4bf4 100644 --- a/web/default/src/features/channels/lib/index.ts +++ b/web/default/src/features/channels/lib/index.ts @@ -22,6 +22,7 @@ export * from './advanced-custom' export * from './channel-capabilities' export * from './channel-config-rules' export * from './channel-config-validation' +export * from './channel-editor-state' export * from './channel-form' export * from './channel-type-config' export * from './channel-utils' diff --git a/web/default/src/features/performance-metrics/types.ts b/web/default/src/features/performance-metrics/types.ts index 64469917..5a58f62f 100644 --- a/web/default/src/features/performance-metrics/types.ts +++ b/web/default/src/features/performance-metrics/types.ts @@ -33,12 +33,38 @@ export type PerformanceGroup = { series: PerformanceSeriesPoint[] } +export type PerformanceAggregate = { + avg_ttft_ms: number + avg_latency_ms: number + success_rate: number + avg_tps: number + series: PerformanceSeriesPoint[] +} + +export type PerformanceCollectionState = + | 'available' + | 'partial' + | 'collection_disabled' + | 'no_samples' + | 'query_failed' + +export type PerformanceCoverage = { + requested_start_at: number + requested_end_at: number + bucket_start_at: number + bucket_end_at: number + bucket_seconds: number + granularity_state: 'known' | 'unknown' | 'mixed' + approximate: boolean +} + export type PerformanceMetricsData = { success: boolean message?: string data: { model_name: string series_schema?: string + summary?: PerformanceAggregate groups: PerformanceGroup[] } } diff --git a/web/default/src/features/pricing/components/model-details-chart-utils.ts b/web/default/src/features/pricing/components/model-details-chart-utils.ts new file mode 100644 index 00000000..02efca43 --- /dev/null +++ b/web/default/src/features/pricing/components/model-details-chart-utils.ts @@ -0,0 +1,116 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import type { LatencyTimePoint, UptimeDayPoint } from '../lib/mock-stats' + +export type LatencyChartDatum = { + time: string + group: string + ttft: number +} + +export type UptimeChartDatum = { + date: string + uptime: number + incidents: number + outage: number +} + +export type UptimeAxisDomain = { + min: number + max: number +} + +export function buildLatencyChartData( + series: LatencyTimePoint[] +): LatencyChartDatum[] { + return series.map((point) => ({ + time: point.timestamp, + group: point.group, + ttft: point.ttft_ms, + })) +} + +export function buildUptimeChartData( + series: UptimeDayPoint[] +): UptimeChartDatum[] { + return series.map((point) => ({ + date: point.date, + uptime: point.uptime_pct, + incidents: point.incidents, + outage: point.outage_minutes, + })) +} + +export function downsampleUptimeSeries( + series: UptimeDayPoint[], + maxPoints: number +): UptimeDayPoint[] { + if (maxPoints <= 0 || series.length <= maxPoints) return series + + return Array.from({ length: maxPoints }, (_, index) => { + const start = Math.floor((index * series.length) / maxPoints) + const end = Math.max( + start + 1, + Math.floor(((index + 1) * series.length) / maxPoints) + ) + let worst = series[start] + for (let cursor = start + 1; cursor < end; cursor += 1) { + if (series[cursor].uptime_pct < worst.uptime_pct) { + worst = series[cursor] + } + } + return worst + }) +} + +export function formatChartAxisTime(iso: string, locale?: string): string { + const date = new Date(iso) + if (Number.isNaN(date.getTime())) return iso + return date.toLocaleTimeString(locale, { + hour: '2-digit', + minute: '2-digit', + }) +} + +export function formatChartTooltipTime(iso: string, locale?: string): string { + const date = new Date(iso) + if (Number.isNaN(date.getTime())) return iso + return date.toLocaleString(locale, { + month: 'short', + day: 'numeric', + hour: '2-digit', + minute: '2-digit', + }) +} + +export function getUptimeAxisDomain( + series: UptimeDayPoint[] +): UptimeAxisDomain { + const values = series + .map((point) => point.uptime_pct) + .filter((value) => Number.isFinite(value)) + if (values.length === 0) return { min: 95, max: 100 } + + const minValue = Math.min(...values) + const maxValue = Math.max(...values) + return { + min: Math.min(95, Math.floor(minValue / 5) * 5), + max: Math.max(100, Math.ceil(maxValue / 5) * 5), + } +} diff --git a/web/default/src/features/pricing/components/model-details-charts.tsx b/web/default/src/features/pricing/components/model-details-charts.tsx index 46f23d91..52780d0d 100644 --- a/web/default/src/features/pricing/components/model-details-charts.tsx +++ b/web/default/src/features/pricing/components/model-details-charts.tsx @@ -25,27 +25,13 @@ import { cn } from '@/lib/utils' import { VCHART_OPTION } from '@/lib/vchart' import { useThemeCustomization } from '@/context/theme-customization-provider' import type { LatencyTimePoint, UptimeDayPoint } from '../lib/mock-stats' - -function formatHourLabel(iso: string): string { - const date = new Date(iso) - const hours = date.getHours() - return `${String(hours).padStart(2, '0')}:00` -} - -function formatDayLabel(date: string): string { - const parsed = new Date(date) - if (date.includes('T')) { - return parsed.toLocaleString(undefined, { - month: 'short', - day: 'numeric', - hour: '2-digit', - }) - } - return parsed.toLocaleDateString(undefined, { - month: 'short', - day: 'numeric', - }) -} +import { + buildLatencyChartData, + buildUptimeChartData, + formatChartAxisTime, + formatChartTooltipTime, + getUptimeAxisDomain, +} from './model-details-chart-utils' function getChartThemeTokens(resolvedTheme: string) { return { @@ -68,17 +54,14 @@ export function LatencyTrendChart(props: { series: LatencyTimePoint[] className?: string }) { - const { t } = useTranslation() + const { t, i18n } = useTranslation() const { resolvedTheme, themeReady } = useChartTheme() const { textColor, gridColor } = getChartThemeTokens(resolvedTheme) const spec = useMemo(() => { if (props.series.length === 0) return null - const data = props.series.map((point) => ({ - time: formatHourLabel(point.timestamp), - group: point.group, - ttft: point.ttft_ms, - })) + + const data = buildLatencyChartData(props.series) return { type: 'line' as const, data: [{ id: 'latency', values: data }], @@ -96,7 +79,10 @@ export function LatencyTrendChart(props: { legends: { visible: false }, tooltip: { mark: { - title: { value: (d: { time: string }) => d.time }, + title: { + value: (d: { time: string }) => + formatChartTooltipTime(d.time, i18n.language), + }, content: [ { key: t('Average TTFT'), @@ -109,6 +95,8 @@ export function LatencyTrendChart(props: { { orient: 'bottom', label: { + formatMethod: (val: number | string) => + formatChartAxisTime(String(val), i18n.language), style: { fill: textColor, fontSize: 10 }, }, tick: { visible: false }, @@ -126,7 +114,7 @@ export function LatencyTrendChart(props: { }, ], } - }, [gridColor, props.series, t, textColor]) + }, [gridColor, i18n.language, props.series, t, textColor]) if (props.series.length === 0) { return ( @@ -166,19 +154,15 @@ export function UptimeTrendChart(props: { series: UptimeDayPoint[] className?: string }) { - const { t } = useTranslation() + const { t, i18n } = useTranslation() const { resolvedTheme, themeReady } = useChartTheme() const { textColor, gridColor } = getChartThemeTokens(resolvedTheme) const spec = useMemo(() => { if (props.series.length === 0) return null - const data = props.series.map((point) => ({ - date: formatDayLabel(point.date), - uptime: point.uptime_pct, - incidents: point.incidents, - outage: point.outage_minutes, - })) + const axisDomain = getUptimeAxisDomain(props.series) + const data = buildUptimeChartData(props.series) return { type: 'line' as const, @@ -204,7 +188,10 @@ export function UptimeTrendChart(props: { }, tooltip: { mark: { - title: { value: (d: { date: string }) => d.date }, + title: { + value: (d: { date: string }) => + formatChartTooltipTime(d.date, i18n.language), + }, content: [ { key: t('Uptime'), @@ -225,6 +212,8 @@ export function UptimeTrendChart(props: { { orient: 'bottom', label: { + formatMethod: (val: number | string) => + formatChartAxisTime(String(val), i18n.language), style: { fill: textColor, fontSize: 10 }, autoLimit: true, }, @@ -232,8 +221,8 @@ export function UptimeTrendChart(props: { }, { orient: 'left', - min: 95, - max: 100, + min: axisDomain.min, + max: axisDomain.max, label: { formatMethod: (val: number | string) => `${val}%`, style: { fill: textColor, fontSize: 10 }, @@ -245,7 +234,7 @@ export function UptimeTrendChart(props: { }, ], } - }, [gridColor, props.series, t, textColor]) + }, [gridColor, i18n.language, props.series, t, textColor]) if (props.series.length === 0) { return ( diff --git a/web/default/src/features/pricing/components/model-details-performance.tsx b/web/default/src/features/pricing/components/model-details-performance.tsx index 7f2d2f8e..7316e3f7 100644 --- a/web/default/src/features/pricing/components/model-details-performance.tsx +++ b/web/default/src/features/pricing/components/model-details-performance.tsx @@ -36,7 +36,11 @@ import { formatThroughput, formatUptimePct, } from '@/features/performance-metrics/lib/format' -import type { PerformanceGroup } from '@/features/performance-metrics/types' +import type { + PerformanceAggregate, + PerformanceGroup, + PerformanceSeriesPoint, +} from '@/features/performance-metrics/types' import { type UptimeDayPoint } from '../lib/mock-stats' import type { PricingModel } from '../types' import { LatencyTrendChart, UptimeTrendChart } from './model-details-charts' @@ -83,59 +87,22 @@ type PerformanceRow = { avg_tps: number } -function toLatencySeries(groups: PerformanceGroup[]) { - const byTs = new Map() - for (const group of groups) { - for (const point of group.series) { - if (point.avg_ttft_ms <= 0) continue - const current = byTs.get(point.ts) ?? [] - current.push(point.avg_ttft_ms) - byTs.set(point.ts, current) - } - } - - return Array.from(byTs.entries()) - .sort(([a], [b]) => a - b) - .map(([ts, values]) => ({ - timestamp: new Date(ts * 1000).toISOString(), - group: 'latency', - ttft_ms: Math.round( - values.reduce((sum, value) => sum + value, 0) / values.length - ), +function toLatencySeriesFromAggregate(summary?: PerformanceAggregate) { + if (!summary) return [] + return summary.series + .filter((point) => point.avg_ttft_ms > 0) + .map((point) => ({ + timestamp: new Date(point.ts * 1000).toISOString(), + group: 'model', + ttft_ms: point.avg_ttft_ms, })) } -function toUptimeSeries(groups: PerformanceGroup[]): UptimeDayPoint[] { - const byTs = new Map() - for (const group of groups) { - for (const point of group.series) { - const current = byTs.get(point.ts) ?? { rates: [], incidents: 0 } - if (Number.isFinite(point.success_rate)) { - current.rates.push(point.success_rate) - if (point.success_rate < 100) current.incidents += 1 - } - byTs.set(point.ts, current) - } - } - return Array.from(byTs.entries()) - .sort(([a], [b]) => a - b) - .map(([ts, value]) => { - const uptime = - value.rates.length > 0 - ? value.rates.reduce((sum, rate) => sum + rate, 0) / - value.rates.length - : 0 - return { - date: new Date(ts * 1000).toISOString(), - uptime_pct: Math.round(uptime * 100) / 100, - incidents: value.incidents, - outage_minutes: 0, - } - }) -} - -function toGroupUptimeSeries(group: PerformanceGroup): UptimeDayPoint[] { - return group.series.map((point) => ({ +function toUptimeSeries( + series: PerformanceSeriesPoint[] | undefined +): UptimeDayPoint[] { + if (!series) return [] + return series.map((point) => ({ date: new Date(point.ts * 1000).toISOString(), uptime_pct: Math.round(point.success_rate * 100) / 100, incidents: point.success_rate < 100 ? 1 : 0, @@ -143,19 +110,7 @@ function toGroupUptimeSeries(group: PerformanceGroup): UptimeDayPoint[] { })) } -function average( - rows: PerformanceRow[], - field: 'avg_ttft_ms' | 'avg_latency_ms' -) { - const values = rows.map((row) => row[field]).filter((value) => value > 0) - if (values.length === 0) return 0 - return Math.round( - values.reduce((sum, value) => sum + value, 0) / values.length - ) -} - export function ModelDetailsPerformance(props: { model: PricingModel }) { - const { t } = useTranslation() const metricsQuery = useQuery({ queryKey: ['perf-metrics', props.model.model_name], queryFn: () => getPerfMetrics(props.model.model_name, 24), @@ -165,6 +120,24 @@ export function ModelDetailsPerformance(props: { model: PricingModel }) { () => metricsQuery.data?.data.groups ?? [], [metricsQuery.data] ) + const summary = metricsQuery.data?.data.summary + + return ( + + ) +} + +export function ModelPerformanceDetailsContent(props: { + groups: PerformanceGroup[] + summary?: PerformanceAggregate + loading?: boolean +}) { + const { t } = useTranslation() + const groups = props.groups const performances = useMemo( () => groups.map((group) => ({ @@ -176,17 +149,35 @@ export function ModelDetailsPerformance(props: { model: PricingModel }) { })), [groups] ) - const latencySeries = useMemo(() => toLatencySeries(groups), [groups]) - const uptimeSeries = useMemo(() => toUptimeSeries(groups), [groups]) + const aggregate = useMemo(() => { + if (props.summary) return props.summary + if (groups.length !== 1) return undefined + const group = groups[0] + return { + avg_ttft_ms: group.avg_ttft_ms, + avg_latency_ms: group.avg_latency_ms, + success_rate: group.success_rate, + avg_tps: group.avg_tps, + series: group.series, + } + }, [groups, props.summary]) + const latencySeries = useMemo( + () => (aggregate ? toLatencySeriesFromAggregate(aggregate) : []), + [aggregate] + ) + const uptimeSeries = useMemo( + () => toUptimeSeries(aggregate?.series), + [aggregate] + ) const uptimeByGroup = useMemo>(() => { const map: Record = {} for (const group of groups) { - map[group.group] = toGroupUptimeSeries(group) + map[group.group] = toUptimeSeries(group.series) } return map }, [groups]) - if (metricsQuery.isLoading || performances.length === 0) { + if (props.loading || performances.length === 0) { return (
{t('Performance data is not yet available for this model.')} @@ -194,28 +185,27 @@ export function ModelDetailsPerformance(props: { model: PricingModel }) { ) } - const tpsValues = performances - .map((p) => p.avg_tps) - .filter((value) => value > 0) - const avgTps = - tpsValues.length > 0 - ? tpsValues.reduce((sum, value) => sum + value, 0) / tpsValues.length - : 0 - const avgLatency = average(performances, 'avg_latency_ms') - const successRates = performances - .map((perf) => perf.success_rate) - .filter((value) => Number.isFinite(value)) - const successRate = - successRates.length > 0 - ? successRates.reduce((sum, value) => sum + value, 0) / - successRates.length - : 0 + const aggregateAvailable = aggregate != null + const avgTps = aggregate?.avg_tps ?? Number.NaN + const avgLatency = aggregate?.avg_latency_ms ?? Number.NaN + const successRate = aggregate?.success_rate ?? Number.NaN const incidentCount = uptimeSeries.reduce((s, p) => s + p.incidents, 0) - let intent: 'default' | 'warning' | 'success' = 'warning' - if (successRate >= 99.9) { - intent = 'success' - } else if (successRate >= 99) { - intent = 'default' + let intent: 'default' | 'warning' | 'success' = 'default' + if (aggregateAvailable) { + if (successRate >= 99.9) { + intent = 'success' + } else if (successRate < 99) { + intent = 'warning' + } + } + let successRateHint = t('N/A') + if (aggregateAvailable) { + successRateHint = + incidentCount > 0 + ? t('{{count}} incidents in the last 24 hours', { + count: incidentCount, + }) + : t('No incidents in the last 24 hours') } const headerCellClass = @@ -239,13 +229,7 @@ export function ModelDetailsPerformance(props: { model: PricingModel }) { icon={HeartPulse} label={t('Success rate')} value={formatUptimePct(successRate)} - hint={ - incidentCount > 0 - ? t('{{count}} incidents in the last 24 hours', { - count: incidentCount, - }) - : t('No incidents in the last 24 hours') - } + hint={successRateHint} intent={intent} />
@@ -296,6 +280,7 @@ export function ModelDetailsPerformance(props: { model: PricingModel }) { diff --git a/web/default/src/features/pricing/components/model-details-uptime-sparkline.tsx b/web/default/src/features/pricing/components/model-details-uptime-sparkline.tsx index 6bdbe747..a0d0d679 100644 --- a/web/default/src/features/pricing/components/model-details-uptime-sparkline.tsx +++ b/web/default/src/features/pricing/components/model-details-uptime-sparkline.tsx @@ -27,6 +27,7 @@ import { } from '@/components/ui/tooltip' import { formatUptimePct } from '@/features/performance-metrics/lib/format' import { aggregateUptime, type UptimeDayPoint } from '../lib/mock-stats' +import { downsampleUptimeSeries } from './model-details-chart-utils' // --------------------------------------------------------------------------- // Uptime sparkline @@ -46,6 +47,7 @@ type UptimeSparklineProps = { series: UptimeDayPoint[] size?: SparklineSize showOverall?: boolean + overallPct?: number emptyLabel?: string className?: string } @@ -75,6 +77,26 @@ export function UptimeSparkline(props: UptimeSparklineProps) { const size = props.size ?? 'md' const showOverall = props.showOverall ?? true + const calculatedOverall = useMemo( + () => + props.series.length > 0 + ? props.series.reduce((sum, point) => sum + point.uptime_pct, 0) / + props.series.length + : 0, + [props.series] + ) + const overall = useMemo( + () => + typeof props.overallPct === 'number' && Number.isFinite(props.overallPct) + ? props.overallPct + : calculatedOverall, + [calculatedOverall, props.overallPct] + ) + const displaySeries = useMemo( + () => downsampleUptimeSeries(props.series, size === 'sm' ? 32 : 48), + [props.series, size] + ) + if (props.series.length === 0) { return ( @@ -83,9 +105,6 @@ export function UptimeSparkline(props: UptimeSparklineProps) { ) } - const overall = - props.series.reduce((s, p) => s + p.uptime_pct, 0) / props.series.length - const containerHeight = size === 'sm' ? 'h-3.5' : 'h-5' const barWidth = size === 'sm' ? 'w-[3px]' : 'w-1' const gap = size === 'sm' ? 'gap-px' : 'gap-[2px]' @@ -99,7 +118,7 @@ export function UptimeSparkline(props: UptimeSparklineProps) { percent: overall.toFixed(2), })} > - {props.series.map((day) => ( + {displaySeries.map((day) => ( {t('Last 30 days uptime')} - +
{statusLabel} diff --git a/web/default/src/features/pricing/lib/tier-expr.test.ts b/web/default/src/features/pricing/lib/tier-expr.test.ts index fc95ec46..991ddcf1 100644 --- a/web/default/src/features/pricing/lib/tier-expr.test.ts +++ b/web/default/src/features/pricing/lib/tier-expr.test.ts @@ -42,6 +42,57 @@ describe('evalExprLocally', () => { assert.deepEqual(result, { cost: 230, matchedTier: 'small', error: null }) }) + test('evaluates time-based pricing expressions with backend time functions', () => { + const result = evalExprLocally( + 'hour("Asia/Shanghai") < 9 || (hour("Asia/Shanghai") >= 12 && hour("Asia/Shanghai") < 14) || hour("Asia/Shanghai") >= 18 ? tier("平常时段 0-9/12-14/18-24", p * 4.5 + c * 13.5 + cr * 0.15) : tier("高峰期 9-12/14-18", p * 9 + c * 27 + cr * 0.3)', + 100, + 10, + emptyExtraTokens + ) + + assert.equal(result.error, null) + assert.ok( + [ + { + cost: 585, + matchedTier: '平常时段 0-9/12-14/18-24', + }, + { + cost: 1170, + matchedTier: '高峰期 9-12/14-18', + }, + ].some( + (expected) => + expected.cost === result.cost && + expected.matchedTier === result.matchedTier + ) + ) + }) + + test('keeps time helpers in backend ranges and falls back to UTC', () => { + const utcFallbackChecks = ['hour', 'minute', 'weekday', 'month', 'day'] + .flatMap((helper) => [ + `${helper}("Invalid/Zone") == ${helper}("UTC")`, + `${helper}("") == ${helper}("UTC")`, + ]) + .join(' && ') + const utcRangeChecks = [ + 'hour("UTC") >= 0 && hour("UTC") <= 23', + 'minute("UTC") >= 0 && minute("UTC") <= 59', + 'weekday("UTC") >= 0 && weekday("UTC") <= 6', + 'month("UTC") >= 1 && month("UTC") <= 12', + 'day("UTC") >= 1 && day("UTC") <= 31', + ].join(' && ') + const result = evalExprLocally( + `${utcFallbackChecks} && ${utcRangeChecks} ? tier("valid", p) : tier("invalid", 999)`, + 42, + 0, + emptyExtraTokens + ) + + assert.deepEqual(result, { cost: 42, matchedTier: 'valid', error: null }) + }) + test('does not expose browser globals or member access', () => { const result = evalExprLocally( 'globalThis.document ? 999 : 1', diff --git a/web/default/src/features/pricing/lib/tier-expr.ts b/web/default/src/features/pricing/lib/tier-expr.ts index 97a00aff..3cfae499 100644 --- a/web/default/src/features/pricing/lib/tier-expr.ts +++ b/web/default/src/features/pricing/lib/tier-expr.ts @@ -277,6 +277,88 @@ const estimatorParser = new Parser({ }, }) +type EstimatorTimeParts = { + hour: number + minute: number + weekday: number + month: number + day: number +} + +type EstimatorTimeFunctions = { + hour: (timezone: string) => number + minute: (timezone: string) => number + weekday: (timezone: string) => number + month: (timezone: string) => number + day: (timezone: string) => number +} + +const ESTIMATOR_WEEKDAYS: Record = { + Sun: 0, + Mon: 1, + Tue: 2, + Wed: 3, + Thu: 4, + Fri: 5, + Sat: 6, +} + +function createEstimatorTimeFunctions(): EstimatorTimeFunctions { + const now = new Date() + const timePartsCache = new Map() + + const getUtcTimeParts = (): EstimatorTimeParts => ({ + hour: now.getUTCHours(), + minute: now.getUTCMinutes(), + weekday: now.getUTCDay(), + month: now.getUTCMonth() + 1, + day: now.getUTCDate(), + }) + + const getTimeParts = (timezone: string): EstimatorTimeParts => { + const normalizedTimezone = timezone.trim() || 'UTC' + const cached = timePartsCache.get(normalizedTimezone) + if (cached) return cached + + try { + const parts = new Intl.DateTimeFormat('en-US-u-ca-gregory-nu-latn', { + timeZone: normalizedTimezone, + hourCycle: 'h23', + hour: '2-digit', + minute: '2-digit', + weekday: 'short', + month: 'numeric', + day: 'numeric', + }).formatToParts(now) + const partValues = Object.fromEntries( + parts.map((part) => [part.type, part.value]) + ) + const result = { + hour: Number(partValues.hour), + minute: Number(partValues.minute), + weekday: ESTIMATOR_WEEKDAYS[partValues.weekday] ?? 0, + month: Number(partValues.month), + day: Number(partValues.day), + } + timePartsCache.set(normalizedTimezone, result) + return result + } catch { + const fallback = timePartsCache.get('UTC') ?? getUtcTimeParts() + timePartsCache.set('UTC', fallback) + timePartsCache.set(normalizedTimezone, fallback) + return fallback + } + } + + return { + hour: (timezone: string) => getTimeParts(timezone).hour, + minute: (timezone: string) => getTimeParts(timezone).minute, + weekday: (timezone: string) => getTimeParts(timezone).weekday, + month: (timezone: string) => getTimeParts(timezone).month, + day: (timezone: string) => getTimeParts(timezone).day, + } +} + function normalizeEstimatorExpression(exprStr: string): string { const body = exprStr.trim().replace(/^v\d+:/, '') let quote = '' @@ -340,6 +422,7 @@ export function evalExprLocally( const cacheCreate1hTokens = extraTokenValues.cacheCreate1hTokens || 0 const len = promptTokens + cacheReadTokens + cacheCreateTokens + cacheCreate1hTokens + const timeFunctions = createEstimatorTimeFunctions() const env: Record = { p: promptTokens, c: completionTokens, @@ -350,6 +433,7 @@ export function evalExprLocally( abs: Math.abs, ceil: Math.ceil, floor: Math.floor, + ...timeFunctions, } for (const field of ESTIMATOR_VARS) { env[field.var] = extraTokenValues[field.stateKey] || 0 diff --git a/web/default/src/features/profile/api.ts b/web/default/src/features/profile/api.ts index aa4a4d2b..b4624736 100644 --- a/web/default/src/features/profile/api.ts +++ b/web/default/src/features/profile/api.ts @@ -17,6 +17,7 @@ along with this program. If not, see . For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues */ import { api, type ApiRequestConfig } from '@/lib/api' +import { selfCustomOAuthUnbindPath, type CustomOAuthBinding } from '@/lib/oauth' import { getTurnstileHeaders } from '@/features/auth/lib/turnstile-request' import type { ApiResponse, @@ -144,12 +145,6 @@ export async function bindWeChat(code: string): Promise { // Custom OAuth Binding APIs // ============================================================================ -export interface CustomOAuthBinding { - provider_id: string - provider_name: string - external_id?: string -} - /** * Get current user's custom OAuth bindings */ @@ -164,9 +159,9 @@ export async function getSelfOAuthBindings(): Promise< * Unbind a custom OAuth provider for current user */ export async function unbindCustomOAuth( - providerId: string + providerId: number ): Promise { - const res = await api.delete(`/api/user/oauth/bindings/${providerId}`) + const res = await api.delete(selfCustomOAuthUnbindPath(providerId)) return res.data } diff --git a/web/default/src/features/profile/components/tabs/account-bindings-tab.tsx b/web/default/src/features/profile/components/tabs/account-bindings-tab.tsx index b0792c05..d806b4a1 100644 --- a/web/default/src/features/profile/components/tabs/account-bindings-tab.tsx +++ b/web/default/src/features/profile/components/tabs/account-bindings-tab.tsx @@ -27,6 +27,8 @@ import { handleOIDCOAuth, handleDiscordOAuth, handleLinuxDOOAuth, + indexCustomOAuthBindings, + type CustomOAuthBinding, } from '@/lib/oauth' import { useDialogs } from '@/hooks/use-dialog' import { useStatus } from '@/hooks/use-status' @@ -35,11 +37,7 @@ import { Separator } from '@/components/ui/separator' import { ConfirmDialog } from '@/components/confirm-dialog' import { StatusBadge } from '@/components/status-badge' import { OAUTH_BIND_STORAGE_KEY } from '@/features/auth/constants' -import { - getSelfOAuthBindings, - unbindCustomOAuth, - type CustomOAuthBinding, -} from '../../api' +import { getSelfOAuthBindings, unbindCustomOAuth } from '../../api' import type { UserProfile, BindingItem } from '../../types' import { EmailBindDialog } from '../dialogs/email-bind-dialog' import { TelegramBindDialog } from '../dialogs/telegram-bind-dialog' @@ -67,10 +65,14 @@ export function AccountBindingsTab({ const [unbindTarget, setUnbindTarget] = useState( null ) + const customBindingIndex = useMemo( + () => indexCustomOAuthBindings(customBindings), + [customBindings] + ) const [unbinding, setUnbinding] = useState(false) const customProviders = status?.custom_oauth_providers as - | Array<{ id: string; name: string }> + | Array<{ id: number; name: string }> | undefined const fetchCustomBindings = useCallback(async () => { @@ -113,7 +115,7 @@ export function AccountBindingsTab({ } } - const handleBindCustomOAuth = (provider: { id: string; name: string }) => { + const handleBindCustomOAuth = (provider: { id: number; name: string }) => { const redirectUrl = `${window.location.origin}/oauth/${provider.id}?bind=true` window.location.href = `/api/oauth/${provider.id}?redirect=${encodeURIComponent(redirectUrl)}` } @@ -315,9 +317,7 @@ export function AccountBindingsTab({

{customProviders.map((provider) => { - const binding = customBindings.find( - (b) => b.provider_id === provider.id - ) + const binding = customBindingIndex.get(provider.id) const isBound = !!binding return (

{isBound - ? binding?.external_id || t('Bound') + ? binding?.provider_user_id || t('Bound') : t('Not bound')}

diff --git a/web/default/src/features/smart-ops/alerts.test.tsx b/web/default/src/features/smart-ops/alerts.test.tsx new file mode 100644 index 00000000..d85570b8 --- /dev/null +++ b/web/default/src/features/smart-ops/alerts.test.tsx @@ -0,0 +1,232 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import { QueryClient, QueryClientProvider } from '@tanstack/react-query' +import { createReactTestEnvironment } from '@/test/react' +import { waitFor, within } from '@testing-library/react' +import assert from 'node:assert/strict' +import { after, before, describe, test } from 'node:test' +import { api } from '@/lib/api' +import { ActiveAlerts } from './alerts' + +const LOAD_ERROR_KEY = 'We could not load active alerts.' +const testEnv = createReactTestEnvironment({ + resources: { + en: { translation: { [LOAD_ERROR_KEY]: LOAD_ERROR_KEY } }, + fr: { + translation: { + [LOAD_ERROR_KEY]: 'Impossible de charger les alertes actives.', + }, + }, + }, +}) + +before(() => testEnv.setup()) + +after(() => testEnv.teardown()) + +function createQueryClient(): QueryClient { + return new QueryClient({ + defaultOptions: { + queries: { retry: false }, + mutations: { retry: false }, + }, + }) +} + +describe('SmartOps active alerts', () => { + test('polls the administrator alert endpoint and renders active host pressure', async () => { + const originalGet = api.get + const urls: string[] = [] + api.get = (async (url) => { + urls.push(String(url)) + return { + data: { + success: true, + data: [ + { + key: 'system_cpu', + status: 'firing', + severity: 'warning', + component: 'system', + node: 'node-a', + current_value: 95.25, + threshold: 90, + observed_at: '2026-08-22T08:00:00Z', + message: 'CPU usage exceeded the threshold', + }, + ], + }, + } + }) as typeof api.get + + const queryClient = createQueryClient() + const view = await testEnv.render( + + + + ) + + try { + await waitFor(() => { + assert.equal(urls[0], '/api/smart-ops/alerts') + const text = view.container.textContent ?? '' + assert.ok(text.includes('CPU usage')) + assert.ok(text.includes('node-a')) + assert.ok(text.includes('95.3%')) + assert.ok(text.includes('90.0%')) + assert.ok(text.includes('Firing')) + }) + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + } + }) + + test('formats percentages with the active locale', async () => { + const originalGet = api.get + await testEnv.i18n.changeLanguage('fr') + api.get = (async () => ({ + data: { + success: true, + data: [ + { + key: 'system_memory', + status: 'firing', + severity: 'warning', + component: 'system', + node: 'node-fr', + current_value: 95.25, + threshold: 90, + observed_at: '2026-08-22T08:00:00Z', + message: 'Memory usage exceeded the threshold', + }, + ], + }, + })) as typeof api.get + + const queryClient = createQueryClient() + const view = await testEnv.render( + + + + ) + + try { + await waitFor(() => { + assert.ok((view.container.textContent ?? '').includes('95,3%')) + }) + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + await testEnv.i18n.changeLanguage('en') + } + }) + + test('shows a healthy empty state when no incident is active', async () => { + const originalGet = api.get + api.get = (async () => ({ + data: { success: true, data: [] }, + })) as typeof api.get + + const queryClient = createQueryClient() + const view = await testEnv.render( + + + + ) + + try { + await waitFor(() => { + assert.ok( + (view.container.textContent ?? '').includes('No active alerts.') + ) + }) + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + } + }) + + test('shows a localized fallback for an unsuccessful response', async () => { + const originalGet = api.get + await testEnv.i18n.changeLanguage('fr') + api.get = (async () => ({ + data: { success: false, data: [] }, + })) as typeof api.get + + const queryClient = createQueryClient() + const view = await testEnv.render( + + + + ) + + try { + await waitFor(() => { + const text = view.container.textContent ?? '' + assert.ok(text.includes('Impossible de charger les alertes actives.')) + assert.ok(!text.includes(LOAD_ERROR_KEY)) + }) + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + await testEnv.i18n.changeLanguage('en') + } + }) + + test('shows a retryable error state when the alert endpoint fails', async () => { + const originalGet = api.get + let requestCount = 0 + api.get = (async () => { + requestCount += 1 + throw new Error('temporary alert endpoint failure') + }) as typeof api.get + + const queryClient = createQueryClient() + const view = await testEnv.render( + + + + ) + + try { + let retryButton: HTMLElement | undefined + await waitFor(() => { + const text = view.container.textContent ?? '' + assert.ok(text.includes('We could not load active alerts.')) + assert.ok(text.includes('temporary alert endpoint failure')) + retryButton = within(view.container).getByRole('button', { + name: 'Retry', + }) + }) + assert.ok(retryButton) + const requestsBeforeRetry = requestCount + await view.click(retryButton) + await waitFor(() => assert.ok(requestCount > requestsBeforeRetry)) + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + } + }) +}) diff --git a/web/default/src/features/smart-ops/alerts.tsx b/web/default/src/features/smart-ops/alerts.tsx new file mode 100644 index 00000000..9f53e8b0 --- /dev/null +++ b/web/default/src/features/smart-ops/alerts.tsx @@ -0,0 +1,251 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import type { ReactElement } from 'react' +import { useQuery } from '@tanstack/react-query' +import { RefreshCw, ShieldCheck, TriangleAlert } from 'lucide-react' +import { useTranslation } from 'react-i18next' +import { formatTimestampRelative, formatTimestampToDate } from '@/lib/format' +import { cn } from '@/lib/utils' +import { Badge } from '@/components/ui/badge' +import { Button } from '@/components/ui/button' +import { + Empty, + EmptyDescription, + EmptyHeader, + EmptyMedia, + EmptyTitle, +} from '@/components/ui/empty' +import { Skeleton } from '@/components/ui/skeleton' +import { + Table, + TableBody, + TableCell, + TableHead, + TableHeader, + TableRow, +} from '@/components/ui/table' +import { ErrorState } from '@/components/error-state' +import { SectionPageLayout } from '@/components/layout' +import { getSmartOpsAlerts } from './api' +import type { SmartOpsAlert } from './types' + +const ALERT_POLL_INTERVAL_MS = 5000 + +const RESOURCE_LABEL: Record = { + system_cpu: 'CPU usage', + system_memory: 'Memory usage', + system_disk: 'Disk usage', +} + +function formatPercent(value: number, locale: string): string { + return `${new Intl.NumberFormat(locale, { + minimumFractionDigits: 1, + maximumFractionDigits: 1, + }).format(value)}%` +} + +function getObservedAtMilliseconds(alert: SmartOpsAlert): number | undefined { + const timestamp = Date.parse(alert.observed_at) + return Number.isNaN(timestamp) ? undefined : timestamp +} + +interface ActiveAlertsTableProps { + alerts: SmartOpsAlert[] +} + +function ActiveAlertsTable(props: ActiveAlertsTableProps): ReactElement { + const { t, i18n } = useTranslation() + + return ( +
+ + + + {t('Resource')} + {t('Status')} + {t('Node')} + + {t('Current Value')} + + + {t('Threshold')} + + + {t('Observed')} + + + + + {props.alerts.map((alert) => { + const observedAt = getObservedAtMilliseconds(alert) + return ( + + +
+ + {t(RESOURCE_LABEL[alert.key] ?? alert.key)} + + + {alert.key} + +
+
+ + {t('Firing')} + + + {alert.node || t('Unknown')} + + + {formatPercent(alert.current_value, i18n.language)} + + + {formatPercent(alert.threshold, i18n.language)} + + + {formatTimestampRelative( + observedAt, + 'milliseconds', + i18n.language + )} + +
+ ) + })} +
+
+
+ ) +} + +export function ActiveAlerts(): ReactElement { + const { t } = useTranslation() + const loadErrorMessage = t('We could not load active alerts.') + const alertsQuery = useQuery({ + queryKey: ['smart-ops', 'active-alerts', loadErrorMessage], + queryFn: async (): Promise => { + const response = await getSmartOpsAlerts() + if (!response.success || !Array.isArray(response.data)) { + throw new Error(response.message || loadErrorMessage) + } + return response.data + }, + retry: false, + refetchInterval: ALERT_POLL_INTERVAL_MS, + }) + + const alerts = alertsQuery.data ?? [] + const loading = alertsQuery.isLoading + const refreshing = alertsQuery.isFetching && !loading + + return ( + + + + {t('Active Alerts')} + + {t('Read only')} + + + + +
+
+
+ + +
+
+

+ {t('Alerts requiring administrator attention')} +

+ 0 ? 'destructive' : 'outline'} + > + {alerts.length} + +
+

+ {t( + 'SmartOps reports sustained host pressure here for manual investigation and repair.' + )} +

+
+
+ +
+ + {loading ? ( +
+ {Array.from({ length: 3 }).map((_, index) => ( + + ))} +
+ ) : alertsQuery.isError ? ( + void alertsQuery.refetch()} + className='min-h-[240px]' + /> + ) : alerts.length === 0 ? ( + + + + + {t('No active alerts.')} + + {t( + 'This process is not currently reporting a sustained CPU, memory, or disk alert.' + )} + + + + ) : ( +
+ +
+ )} +
+
+
+ ) +} diff --git a/web/default/src/features/smart-ops/api.ts b/web/default/src/features/smart-ops/api.ts new file mode 100644 index 00000000..2da83b80 --- /dev/null +++ b/web/default/src/features/smart-ops/api.ts @@ -0,0 +1,97 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import { api } from '@/lib/api' +import { MAX_PERFORMANCE_LIMIT } from './lib/filters' +import type { + ChannelPerformanceData, + ChannelPerformanceQuery, + ChannelPerformanceResponse, + ModelPerformanceData, + ModelPerformanceDetailData, + ModelPerformanceDetailResponse, + ModelPerformanceQuery, + ModelPerformanceResponse, + SmartOpsAlertsResponse, +} from './types' + +export async function getSmartOpsAlerts(): Promise { + const response = await api.get( + '/api/smart-ops/alerts' + ) + return response.data +} + +export async function getChannelPerformance( + query: ChannelPerformanceQuery +): Promise { + const response = await api.get( + '/api/smart-ops/channel-performance', + { + params: { + hours: query.hours, + limit: MAX_PERFORMANCE_LIMIT, + channel_id: query.channelId, + model: query.model || undefined, + group: query.group || undefined, + }, + } + ) + return response.data.data +} + +export async function getChannelPerformanceDetail( + channelId: number +): Promise { + const response = await api.get( + '/api/smart-ops/channel-performance/detail', + { + params: { channel_id: channelId }, + } + ) + return response.data.data +} + +export async function getModelPerformanceDetail( + modelName: string +): Promise { + const response = await api.get( + '/api/smart-ops/model-performance/detail', + { + params: { model: modelName }, + } + ) + return response.data.data +} + +export async function getModelPerformance( + query: ModelPerformanceQuery +): Promise { + const response = await api.get( + '/api/smart-ops/model-performance', + { + params: { + hours: query.hours, + limit: MAX_PERFORMANCE_LIMIT, + model: query.model || undefined, + group: query.group || undefined, + }, + } + ) + return response.data.data +} diff --git a/web/default/src/features/smart-ops/index.test.tsx b/web/default/src/features/smart-ops/index.test.tsx new file mode 100644 index 00000000..c3066abd --- /dev/null +++ b/web/default/src/features/smart-ops/index.test.tsx @@ -0,0 +1,561 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import { act } from 'react' +import { QueryClient, QueryClientProvider } from '@tanstack/react-query' +import { createReactTestEnvironment } from '@/test/react' +import { waitFor } from '@testing-library/react' +import assert from 'node:assert/strict' +import { after, before, describe, test } from 'node:test' +import { api } from '@/lib/api' +import { formatQuota } from '@/lib/format' +import { ProductionPerformance } from './index' +import type { ChannelPerformanceData } from './types' + +const testEnv = createReactTestEnvironment() + +before(() => testEnv.setup()) + +after(() => testEnv.teardown()) + +const emptyPerformanceData: ChannelPerformanceData = { + storage_mode: 'legacy_log', + source: 'log_db', + partial: true, + quality_flags: ['legacy'], + time_range: { start_at: 1, end_at: 3601, hours: 1 }, + summary: { + channel_count: 0, + observed_count: 0, + consume_log_count: 0, + error_log_count: 0, + consumed_quota: 0, + retry_log_count: 0, + latency_sample_count: 0, + observed_success_rate: null, + avg_logged_latency_ms: null, + last_observed_at: 0, + }, + items: [], + truncated: false, + generated_at: 3601, +} + +const performanceDataWithErrors: ChannelPerformanceData = { + ...emptyPerformanceData, + summary: { + ...emptyPerformanceData.summary, + channel_count: 1, + observed_count: 10, + consume_log_count: 7, + error_log_count: 3, + consumed_quota: 750000, + observed_success_rate: 70, + last_observed_at: 3600, + }, + items: [ + { + channel_id: 12, + channel_name: 'Test channel', + channel_type: 1, + channel_status: 1, + model_name: 'test-model', + effective_group: 'default', + observed_count: 10, + consume_log_count: 7, + error_log_count: 3, + consumed_quota: 750000, + retry_log_count: 0, + latency_sample_count: 7, + observed_success_rate: 70, + avg_logged_latency_ms: 1000, + last_observed_at: 3600, + probe_latency_ms: 100, + probe_test_time: 3500, + quality_flags: ['legacy'], + }, + ], +} + +const performanceDataWithDisabledErrorLogs: ChannelPerformanceData = { + ...performanceDataWithErrors, + quality_flags: [ + 'legacy', + 'partial', + 'weak_correlation', + 'heuristic_outcome', + 'coarse_latency', + 'probe_separate', + 'non_attempt_logs_excluded', + 'incomplete_errors_possible', + 'error_logs_disabled', + ], + summary: { + ...performanceDataWithErrors.summary, + error_log_count: 0, + observed_success_rate: null, + }, + items: performanceDataWithErrors.items.map((item) => ({ + ...item, + error_log_count: 0, + observed_success_rate: null, + quality_flags: [ + ...item.quality_flags, + 'incomplete_errors_possible', + 'error_logs_disabled', + ], + })), +} + +const performanceDataWithSortableRows: ChannelPerformanceData = { + ...performanceDataWithErrors, + items: [ + { + ...performanceDataWithErrors.items[0], + channel_id: 21, + channel_name: 'Healthy channel', + observed_count: 20, + consume_log_count: 20, + error_log_count: 0, + consumed_quota: 1000000, + observed_success_rate: 100, + avg_logged_latency_ms: 300, + retry_log_count: 0, + probe_latency_ms: 80, + }, + { + ...performanceDataWithErrors.items[0], + channel_id: 22, + channel_name: 'Problem channel', + observed_count: 12, + consume_log_count: 4, + error_log_count: 8, + consumed_quota: 200000, + observed_success_rate: 33.333, + avg_logged_latency_ms: 5000, + retry_log_count: 6, + probe_latency_ms: 1200, + }, + { + ...performanceDataWithErrors.items[0], + channel_id: 23, + channel_name: 'Unknown channel', + observed_count: 2, + consume_log_count: 2, + error_log_count: 0, + consumed_quota: 0, + observed_success_rate: null, + avg_logged_latency_ms: null, + retry_log_count: null, + probe_latency_ms: null, + }, + ], +} + +const channelDetailData: ChannelPerformanceData = { + ...performanceDataWithErrors, + quality_flags: ['error_logs_disabled'], + time_range: { start_at: 1, end_at: 86401, hours: 24 }, + summary: { + ...performanceDataWithErrors.summary, + observed_count: 15, + consume_log_count: 12, + error_log_count: 3, + consumed_quota: 900000, + observed_success_rate: 80, + last_observed_at: 7200, + }, + items: [ + performanceDataWithErrors.items[0], + { + ...performanceDataWithErrors.items[0], + model_name: 'second-model', + effective_group: 'premium', + observed_count: 5, + consume_log_count: 5, + error_log_count: 0, + consumed_quota: 150000, + observed_success_rate: 100, + }, + ], +} + +describe('ProductionPerformance manual queries', () => { + test('requests data only after explicit user actions', async () => { + const originalGet = api.get + let requestCount = 0 + api.get = (async () => { + requestCount += 1 + return { data: { success: true, data: emptyPerformanceData } } + }) as typeof api.get + + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false }, + mutations: { retry: false }, + }, + }) + const view = await testEnv.render( + + + + ) + + try { + await act(async () => { + await new Promise((resolve) => setTimeout(resolve, 0)) + }) + assert.equal(requestCount, 0, 'mounting the page must not query logs') + + const findButton = (label: string) => { + const button = Array.from( + view.container.querySelectorAll('button') + ).find((candidate) => candidate.textContent?.trim() === label) + assert.ok(button, `expected ${label} button`) + return button + } + + await view.click(findButton('Apply filters')) + await waitFor(() => assert.equal(requestCount, 1)) + + await view.click(findButton('Apply filters')) + await waitFor(() => assert.equal(requestCount, 2)) + + await act(async () => { + window.dispatchEvent(new Event('focus')) + window.dispatchEvent(new Event('online')) + await new Promise((resolve) => setTimeout(resolve, 0)) + }) + assert.equal(requestCount, 2, 'focus and reconnect must not query logs') + + await view.click(findButton('Refresh')) + await waitFor(() => assert.equal(requestCount, 3)) + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + } + }) + + test('shows error counts and consumed quota in the channel performance table', async () => { + const originalGet = api.get + api.get = (async () => ({ + data: { success: true, data: performanceDataWithErrors }, + })) as typeof api.get + + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false }, + mutations: { retry: false }, + }, + }) + const view = await testEnv.render( + + + + ) + + try { + const applyButton = Array.from( + view.container.querySelectorAll('button') + ).find((candidate) => candidate.textContent?.trim() === 'Apply filters') + assert.ok(applyButton, 'expected Apply filters button') + + await view.click(applyButton) + await waitFor(() => { + assert.ok( + Array.from( + view.container.querySelectorAll('[data-slot="card-description"]') + ).some((node) => node.textContent?.trim() === 'Recorded error logs') + ) + }) + + const errorCardLabel = Array.from( + view.container.querySelectorAll('[data-slot="card-description"]') + ).find((node) => node.textContent?.trim() === 'Recorded error logs') + assert.ok(errorCardLabel, 'expected error summary card') + assert.equal( + errorCardLabel.parentElement?.querySelector('[data-slot="card-title"]') + ?.textContent, + '3' + ) + + const headers = Array.from(view.container.querySelectorAll('thead th')) + const errorColumnIndex = headers.findIndex( + (header) => header.textContent?.trim() === 'Recorded error logs' + ) + assert.ok(errorColumnIndex >= 0, 'expected error count table column') + const rowCells = view.container.querySelectorAll( + 'tbody tr:first-child td' + ) + assert.equal(rowCells.item(errorColumnIndex).textContent?.trim(), '3') + const consumedQuotaColumnIndex = headers.findIndex( + (header) => header.textContent?.trim() === 'Consumed quota' + ) + assert.ok(consumedQuotaColumnIndex >= 0, 'expected consumed quota column') + assert.equal( + rowCells.item(consumedQuotaColumnIndex).textContent?.trim(), + formatQuota(750000) + ) + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + } + }) + + test('keeps recorded errors visible and warns when error logging is disabled', async () => { + const originalGet = api.get + api.get = (async () => ({ + data: { + success: true, + data: performanceDataWithDisabledErrorLogs, + }, + })) as typeof api.get + + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false }, + mutations: { retry: false }, + }, + }) + const view = await testEnv.render( + + + + ) + + try { + const applyButton = Array.from( + view.container.querySelectorAll('button') + ).find((candidate) => candidate.textContent?.trim() === 'Apply filters') + assert.ok(applyButton, 'expected Apply filters button') + + await view.click(applyButton) + await waitFor(() => { + assert.ok( + Array.from( + view.container.querySelectorAll('[data-slot="alert-title"]') + ).some( + (node) => + node.textContent?.trim() === 'Error log collection is disabled' + ) + ) + }) + + const errorCardLabel = Array.from( + view.container.querySelectorAll('[data-slot="card-description"]') + ).find((node) => node.textContent?.trim() === 'Recorded error logs') + assert.ok(errorCardLabel, 'expected recorded error summary card') + assert.equal( + errorCardLabel.parentElement?.querySelector('[data-slot="card-title"]') + ?.textContent, + '0' + ) + + const successCardLabel = Array.from( + view.container.querySelectorAll('[data-slot="card-description"]') + ).find((node) => node.textContent?.trim() === 'Estimated success rate') + assert.ok(successCardLabel, 'expected success-rate summary card') + assert.equal( + successCardLabel.parentElement?.querySelector( + '[data-slot="card-title"]' + )?.textContent, + 'N/A' + ) + + assert.ok( + Array.from(view.container.querySelectorAll('[data-slot="badge"]')).some( + (node) => node.textContent?.trim() === 'Error logs disabled' + ), + 'critical log collection state must not be hidden by badge truncation' + ) + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + } + }) + + test('provides a semantic details button without making the table row tabbable', async () => { + const originalGet = api.get + api.get = (async () => ({ + data: { success: true, data: performanceDataWithErrors }, + })) as typeof api.get + + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false }, + mutations: { retry: false }, + }, + }) + const view = await testEnv.render( + + + + ) + + try { + const applyButton = Array.from( + view.container.querySelectorAll('button') + ).find((candidate) => candidate.textContent?.trim() === 'Apply filters') + assert.ok(applyButton, 'expected Apply filters button') + await view.click(applyButton) + + const detailsButton = await waitFor(() => { + const button = Array.from( + view.container.querySelectorAll('tbody button') + ).find((candidate) => candidate.textContent?.trim() === 'Details') + assert.ok(button, 'expected semantic details button') + return button + }) + assert.equal( + detailsButton.getAttribute('aria-label'), + 'View details: Test channel · test-model' + ) + assert.equal( + detailsButton.closest('tr')?.getAttribute('tabindex'), + null, + 'the row must not compete with the semantic button in keyboard order' + ) + assert.equal(detailsButton.tagName, 'BUTTON') + assert.equal(detailsButton.getAttribute('type'), 'button') + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + } + }) + + test('loads a channel-scoped 24-hour detail only when the detail sheet opens', async () => { + const originalGet = api.get + const urls: string[] = [] + api.get = (async (url, config) => { + urls.push(String(url)) + if (url === '/api/smart-ops/channel-performance/detail') { + assert.deepEqual(config?.params, { channel_id: 12 }) + return { data: { success: true, data: channelDetailData } } + } + return { data: { success: true, data: performanceDataWithErrors } } + }) as typeof api.get + + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false }, + mutations: { retry: false }, + }, + }) + const view = await testEnv.render( + + + + ) + + try { + assert.deepEqual(urls, []) + const applyButton = Array.from( + view.container.querySelectorAll('button') + ).find((candidate) => candidate.textContent?.trim() === 'Apply filters') + assert.ok(applyButton) + await view.click(applyButton) + await waitFor(() => assert.equal(urls.length, 1)) + assert.equal(urls[0], '/api/smart-ops/channel-performance') + + const detailsButton = await waitFor(() => { + const button = Array.from( + view.container.querySelectorAll('tbody button') + ).find((candidate) => candidate.textContent?.trim() === 'Details') + assert.ok(button) + return button + }) + await view.click(detailsButton) + + await waitFor(() => { + assert.equal(urls.length, 2) + assert.equal(urls[1], '/api/smart-ops/channel-performance/detail') + }) + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + } + }) + + test('exposes an accessible sort menu for every performance metric', async () => { + const originalGet = api.get + api.get = (async () => ({ + data: { success: true, data: performanceDataWithSortableRows }, + })) as typeof api.get + + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false }, + mutations: { retry: false }, + }, + }) + const view = await testEnv.render( + + + + ) + + try { + const applyButton = Array.from( + view.container.querySelectorAll('button') + ).find((candidate) => candidate.textContent?.trim() === 'Apply filters') + assert.ok(applyButton, 'expected Apply filters button') + await view.click(applyButton) + + const sortableLabels = [ + 'Observed log events', + 'Recorded error logs', + 'Consumed quota', + 'Estimated success rate', + 'Logged latency', + 'Retries', + 'Probe latency', + ] + const headerButtons = await waitFor(() => { + const buttons = Array.from( + view.container.querySelectorAll('thead button') + ) + for (const label of sortableLabels) { + assert.ok( + buttons.some((button) => button.textContent?.trim() === label), + `expected sortable ${label} header` + ) + } + return buttons + }) + + const successRateHeader = headerButtons.find( + (button) => button.textContent?.trim() === 'Estimated success rate' + ) + assert.ok(successRateHeader, 'expected success-rate sort trigger') + for (const button of headerButtons) { + if (!sortableLabels.includes(button.textContent?.trim() ?? '')) continue + assert.equal(button.getAttribute('aria-haspopup'), 'menu') + assert.equal(button.closest('th')?.getAttribute('aria-sort'), 'none') + } + } finally { + api.get = originalGet + queryClient.clear() + await view.unmount() + } + }) +}) diff --git a/web/default/src/features/smart-ops/index.tsx b/web/default/src/features/smart-ops/index.tsx new file mode 100644 index 00000000..ebd94036 --- /dev/null +++ b/web/default/src/features/smart-ops/index.tsx @@ -0,0 +1,1016 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import { useMemo, useState } from 'react' +import { useMutation } from '@tanstack/react-query' +import { + AlertTriangle, + ArrowDown, + ArrowUp, + Clock3, + DatabaseZap, + HeartPulse, + RefreshCw, + Search, + ShieldCheck, +} from 'lucide-react' +import { useTranslation } from 'react-i18next' +import { formatQuota } from '@/lib/format' +import { Alert, AlertDescription, AlertTitle } from '@/components/ui/alert' +import { Badge } from '@/components/ui/badge' +import { Button } from '@/components/ui/button' +import { + Card, + CardAction, + CardDescription, + CardHeader, + CardTitle, +} from '@/components/ui/card' +import { + DropdownMenu, + DropdownMenuContent, + DropdownMenuGroup, + DropdownMenuItem, + DropdownMenuTrigger, +} from '@/components/ui/dropdown-menu' +import { + Empty, + EmptyDescription, + EmptyHeader, + EmptyMedia, + EmptyTitle, +} from '@/components/ui/empty' +import { Field, FieldGroup, FieldLabel } from '@/components/ui/field' +import { Input } from '@/components/ui/input' +import { + Sheet, + SheetContent, + SheetDescription, + SheetHeader, + SheetTitle, +} from '@/components/ui/sheet' +import { Skeleton } from '@/components/ui/skeleton' +import { + Table, + TableBody, + TableCell, + TableHead, + TableHeader, + TableRow, +} from '@/components/ui/table' +import { ErrorState } from '@/components/error-state' +import { SectionPageLayout } from '@/components/layout' +import { + formatLatency, + formatUptimePct, +} from '@/features/performance-metrics/lib/format' +import { getChannelPerformance, getChannelPerformanceDetail } from './api' +import { getChannelDetailObservation } from './lib/channel-detail' +import { + DEFAULT_FILTERS, + MAX_PERFORMANCE_HOURS, + normalizeFilters, + toQuery, +} from './lib/filters' +import { formatLegacyLatency } from './lib/format' +import { + sortChannelPerformanceItems, + type ChannelPerformanceSortKey, + type ChannelPerformanceSortState, +} from './lib/sort' +import { getAriaSort } from './lib/sort-direction' +import { SortDirectionIcon } from './lib/sort-direction-icon' +import type { ChannelPerformanceData, ChannelPerformanceItem } from './types' + +const priorityQualityFlags = [ + 'error_logs_disabled', + 'consume_logs_disabled', + 'retry_backfill_pending', + 'retry_metrics_unavailable', + 'time_range_clamped', + 'truncated', + 'incomplete_errors_possible', +] + +function getVisibleQualityFlags(flags: string[]) { + const flagSet = new Set(flags) + const priorityFlags = priorityQualityFlags.filter((flag) => flagSet.has(flag)) + const otherFlags = flags.filter( + (flag) => !priorityQualityFlags.includes(flag) + ) + return [ + ...priorityFlags, + ...otherFlags.slice(0, Math.max(0, 4 - priorityFlags.length)), + ] +} + +function statusBadge( + status: number | null, + enabledLabel: string, + disabledLabel: string +) { + if (status == null) return — + return status === 1 ? ( + {enabledLabel} + ) : ( + {disabledLabel} + ) +} + +export function ProductionPerformance() { + const { t, i18n } = useTranslation() + const [draft, setDraft] = useState(DEFAULT_FILTERS) + const [filters, setFilters] = useState(DEFAULT_FILTERS) + const [selected, setSelected] = useState(null) + const [sortState, setSortState] = useState(null) + const queryParams = useMemo(() => toQuery(filters), [filters]) + const numberFormatter = useMemo( + () => new Intl.NumberFormat(i18n.language), + [i18n.language] + ) + const dateFormatter = useMemo( + () => + new Intl.DateTimeFormat(i18n.language, { + dateStyle: 'medium', + timeStyle: 'medium', + }), + [i18n.language] + ) + + const performanceQuery = useMutation({ + mutationKey: ['smart-ops', 'channel-performance'], + mutationFn: getChannelPerformance, + retry: false, + }) + const detailQuery = useMutation({ + mutationKey: ['smart-ops', 'channel-performance-detail'], + mutationFn: getChannelPerformanceDetail, + retry: false, + }) + + const data = performanceQuery.data + const selectedDetailObservation = selected + ? getChannelDetailObservation(selected, detailQuery.data) + : null + const sortedItems = useMemo( + () => sortChannelPerformanceItems(data?.items ?? [], sortState), + [data?.items, sortState] + ) + const latestObservedAt = data?.summary.last_observed_at ?? 0 + const retryBackfillPending = + data?.quality_flags.includes('retry_backfill_pending') ?? false + const retryMetricsUnavailable = + data?.quality_flags.includes('retry_metrics_unavailable') ?? false + const errorLogsDisabled = + data?.quality_flags.includes('error_logs_disabled') ?? false + + const openDetails = (item: ChannelPerformanceItem) => { + detailQuery.reset() + setSelected(item) + detailQuery.mutate(item.channel_id) + } + + const formatTimestamp = (timestamp: number | null | undefined) => { + if (!timestamp) return '—' + return dateFormatter.format(new Date(timestamp * 1000)) + } + + const qualityLabel = (flag: string) => { + const labels: Record = { + legacy: t('Legacy log'), + partial: t('Partial data'), + weak_correlation: t('Weak correlation'), + heuristic_outcome: t('Heuristic outcome'), + coarse_latency: t('Coarse latency'), + probe_separate: t('Probe data is separate'), + non_attempt_logs_excluded: t('Known non-attempt logs excluded'), + time_range_clamped: t('Time range was limited'), + truncated: t('Results truncated'), + retry_backfill_pending: t('Retry marker backfill pending'), + retry_metrics_unavailable: t('Retry metrics unavailable'), + consume_logs_disabled: t('Consumption logs disabled'), + error_logs_disabled: t('Error logs disabled'), + incomplete_errors_possible: t('Errors may be incomplete'), + metadata_missing: t('Channel metadata missing'), + probe_unavailable: t('Probe unavailable'), + } + return labels[flag] ?? flag + } + + let summarySuccessRate = '—' + if (data) { + summarySuccessRate = + data.summary.observed_success_rate == null + ? t('N/A') + : formatUptimePct(data.summary.observed_success_rate) + } + const showPerformanceError = + !performanceQuery.isPending && performanceQuery.isError + const showPerformanceIdle = + !performanceQuery.isPending && + !performanceQuery.isError && + performanceQuery.isIdle + const showPerformanceEmpty = + !performanceQuery.isPending && + !performanceQuery.isError && + !performanceQuery.isIdle && + !data?.items.length + const showPerformanceTable = + !performanceQuery.isPending && + !performanceQuery.isError && + !performanceQuery.isIdle && + Boolean(data?.items.length) + const showDetailError = !detailQuery.isPending && detailQuery.isError + const showDetailData = + !detailQuery.isPending && !detailQuery.isError && detailQuery.data != null + + return ( + + + + {t('Channel performance')} + {t('Read only')} + {t('Legacy log')} + + + + + + +
+ + + + {(retryBackfillPending || retryMetricsUnavailable) && ( + + + )} + + {errorLogsDisabled && ( + + + )} + +
{ + event.preventDefault() + const normalizedDraft = normalizeFilters(draft) + const nextQuery = toQuery(normalizedDraft) + setDraft(normalizedDraft) + setFilters(normalizedDraft) + performanceQuery.mutate(nextQuery) + }} + > + + + + {t('Time range')} ({t('hours')}, 1–{MAX_PERFORMANCE_HOURS}) + + + setDraft((current) => ({ + ...current, + hours: event.target.value, + })) + } + /> + + + + {t('Channel ID')} + + + setDraft((current) => ({ + ...current, + channelId: event.target.value, + })) + } + /> + + + + {t('Model')} + + + setDraft((current) => ({ + ...current, + model: event.target.value, + })) + } + /> + + + + {t('Group')} + + + setDraft((current) => ({ + ...current, + group: event.target.value, + })) + } + /> + + + +
+ +
+ + + + + +
+ + {performanceQuery.isPending && } + {showPerformanceError && ( + performanceQuery.mutate(queryParams)} + /> + )} + {showPerformanceIdle && ( + + + + + + {t('Channel performance is not loaded automatically')} + + + {t( + 'Select Apply filters or Refresh to query the current time range.' + )} + + + + )} + {showPerformanceEmpty && ( + + + + + {t('No recorded channel logs')} + + {t( + 'Try a wider time range or check whether consumption and error logging are enabled.' + )} + + + + )} + {showPerformanceTable && data && ( +
+
+ + {t('Channel performance')} + + + {t('{{count}} rows', { count: data.items.length })} + +
+ {getVisibleQualityFlags(data.quality_flags).map((flag) => ( + + {qualityLabel(flag)} + + ))} +
+
+ + + + {t('Channel')} + {t('Model')} + {t('Group')} + + + + + + + + {t('Last observed')} + {t('Actions')} + + + + {sortedItems.map((item) => ( + + +
+
+
+ {item.channel_name} +
+
+ #{item.channel_id} +
+
+ {statusBadge( + item.channel_status, + t('Enabled'), + t('Disabled') + )} +
+
+ + {item.model_name} + + {item.effective_group || '—'} + + {numberFormatter.format(item.observed_count)} + + + {numberFormatter.format(item.error_log_count)} + + + {formatQuota(item.consumed_quota)} + + + {item.observed_success_rate == null + ? t('N/A') + : formatUptimePct(item.observed_success_rate)} + + + {formatLegacyLatency(item.avg_logged_latency_ms)} + + + {item.retry_log_count == null + ? t('N/A') + : numberFormatter.format(item.retry_log_count)} + + + {formatLatency(item.probe_latency_ms ?? Number.NaN)} + + + {formatTimestamp(item.last_observed_at)} + + + + +
+ ))} +
+
+ {data.truncated && ( +
+ {t( + 'The table was truncated to {{count}} rows. Summary cards still cover all matching eligible logs.', + { count: data.items.length } + )} +
+ )} +
+ )} + + {data && ( +
+ {t('Generated at {{time}}', { + time: formatTimestamp(data.generated_at), + })} +
+ )} + { + if (!open) { + setSelected(null) + detailQuery.reset() + } + }} + > + + + {t('Channel performance details')} + + {selected + ? `${selected.channel_name} · #${selected.channel_id}` + : ''} + + + {selected && ( +
+ +
+
+ {t('Data quality')} +
+
+ {selectedDetailObservation?.qualityFlags.length ? ( + selectedDetailObservation.qualityFlags.map((flag) => ( + + {qualityLabel(flag)} + + )) + ) : ( + {t('N/A')} + )} +
+
+ + + {detailQuery.isPending && ( + + )} + {showDetailError && ( + detailQuery.mutate(selected.channel_id)} + /> + )} + {showDetailData && detailQuery.data && ( + + )} +
+ )} +
+
+
+
+
+ ) +} + +function SortablePerformanceHead({ + sortKey, + title, + sortState, + onSort, +}: { + sortKey: ChannelPerformanceSortKey + title: string + sortState: ChannelPerformanceSortState + onSort: (sortState: ChannelPerformanceSortState) => void +}) { + const { t } = useTranslation() + const direction = sortState?.key === sortKey ? sortState.direction : null + const ariaSort = getAriaSort(direction) + + return ( + +
+ + + } + > + {title} + + + + + onSort({ key: sortKey, direction: 'asc' })} + > + + onSort({ key: sortKey, direction: 'desc' })} + > + + + + +
+
+ ) +} + +function MetricCard({ + icon: Icon, + label, + value, + loading, +}: { + icon: typeof DatabaseZap + label: string + value: string + loading: boolean +}) { + return ( + + + {label} + + {loading ? : value} + + +
+
+
+
+
+ ) +} + +function PerformanceTableSkeleton() { + return ( + + ) +} + +function ChannelPerformanceDetail({ + data, + numberFormatter, + formatTimestamp, +}: { + data: ChannelPerformanceData + numberFormatter: Intl.NumberFormat + formatTimestamp: (timestamp: number | null | undefined) => string +}) { + const { t } = useTranslation() + const summary = data.summary + + return ( +
+
+
+
{t('Last 24 hours')}
+
+ {t('Channel models and groups from eligible production logs')} +
+
+ + {t('{{count}} rows', { count: data.items.length })} + +
+ + {data.items.length === 0 ? ( + + + + + {t('No channel model performance')} + + {t( + 'No eligible production logs were recorded for this channel in the last 24 hours.' + )} + + + + ) : ( +
+ + + + {t('Model')} + {t('Group')} + + {t('Observed log events')} + + + {t('Recorded error logs')} + + + {t('Estimated success rate')} + + + {t('Consumed quota')} + + {t('Retries')} + + {t('Logged latency')} + + {t('Last observed')} + + + + {data.items.map((item) => ( + + + {item.model_name} + + {item.effective_group || '—'} + + {numberFormatter.format(item.observed_count)} + + + {numberFormatter.format(item.error_log_count)} + + + {item.observed_success_rate == null + ? t('N/A') + : formatUptimePct(item.observed_success_rate)} + + + {formatQuota(item.consumed_quota)} + + + {item.retry_log_count == null + ? t('N/A') + : numberFormatter.format(item.retry_log_count)} + + + {formatLegacyLatency(item.avg_logged_latency_ms)} + + + {formatTimestamp(item.last_observed_at)} + + + ))} + +
+
+ )} + {data.truncated && ( +
+ {t( + 'The detail table was truncated to {{count}} rows. The summary still covers all matching eligible logs.', + { count: data.items.length } + )} +
+ )} +
+ ) +} + +function ChannelPerformanceDetailSkeleton() { + return ( +
+ +
+ {Array.from({ length: 6 }).map((_, index) => ( + + ))} +
+ +
+ ) +} + +function DetailGrid({ rows }: { rows: [string, string][] }) { + return ( +
+ {rows.map(([label, value]) => ( +
+
{label}
+
+ {value} +
+
+ ))} +
+ ) +} diff --git a/web/default/src/features/smart-ops/lib/channel-detail.test.ts b/web/default/src/features/smart-ops/lib/channel-detail.test.ts new file mode 100644 index 00000000..4aa2242a --- /dev/null +++ b/web/default/src/features/smart-ops/lib/channel-detail.test.ts @@ -0,0 +1,76 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import assert from 'node:assert/strict' +import { describe, test } from 'node:test' +import type { ChannelPerformanceData, ChannelPerformanceItem } from '../types' +import { getChannelDetailObservation } from './channel-detail' + +const selected: ChannelPerformanceItem = { + channel_id: 12, + channel_name: 'Test channel', + channel_type: 1, + channel_status: 1, + model_name: 'test-model', + effective_group: 'default', + observed_count: 10, + consume_log_count: 7, + error_log_count: 3, + consumed_quota: 750000, + retry_log_count: 0, + latency_sample_count: 7, + observed_success_rate: 70, + avg_logged_latency_ms: 1000, + last_observed_at: 3600, + probe_latency_ms: 100, + probe_test_time: 3500, + quality_flags: ['legacy'], +} + +const detail: ChannelPerformanceData = { + storage_mode: 'legacy_log', + source: 'log_db', + partial: true, + quality_flags: ['legacy', 'partial'], + time_range: { start_at: 1, end_at: 3601, hours: 1 }, + summary: { + channel_count: 1, + observed_count: 4, + consume_log_count: 3, + error_log_count: 1, + consumed_quota: 100, + retry_log_count: 0, + latency_sample_count: 3, + observed_success_rate: 75, + avg_logged_latency_ms: 1000, + last_observed_at: 7200, + }, + items: [], + truncated: false, + generated_at: 7200, +} + +describe('channel detail observation', () => { + test('uses channel-scoped detail evidence instead of the clicked list row', () => { + const observation = getChannelDetailObservation(selected, detail) + + assert.equal(observation.lastObservedAt, detail.summary.last_observed_at) + assert.deepEqual(observation.qualityFlags, detail.quality_flags) + assert.equal(observation.probeLatencyMs, selected.probe_latency_ms) + }) +}) diff --git a/web/default/src/features/smart-ops/lib/channel-detail.ts b/web/default/src/features/smart-ops/lib/channel-detail.ts new file mode 100644 index 00000000..330cfbb5 --- /dev/null +++ b/web/default/src/features/smart-ops/lib/channel-detail.ts @@ -0,0 +1,36 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import type { ChannelPerformanceData, ChannelPerformanceItem } from '../types' + +export type ChannelDetailObservation = { + lastObservedAt: number | null + qualityFlags: string[] + probeLatencyMs: number | null +} + +export function getChannelDetailObservation( + selected: ChannelPerformanceItem, + detail: ChannelPerformanceData | undefined +): ChannelDetailObservation { + return { + lastObservedAt: detail?.summary.last_observed_at ?? null, + qualityFlags: detail?.quality_flags ?? [], + probeLatencyMs: selected.probe_latency_ms, + } +} diff --git a/web/default/src/features/smart-ops/lib/filters.test.ts b/web/default/src/features/smart-ops/lib/filters.test.ts new file mode 100644 index 00000000..1ddcdd4a --- /dev/null +++ b/web/default/src/features/smart-ops/lib/filters.test.ts @@ -0,0 +1,87 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import assert from 'node:assert/strict' +import { describe, test } from 'node:test' +import { + DEFAULT_FILTERS, + DEFAULT_MODEL_FILTERS, + DEFAULT_PERFORMANCE_HOURS, + MAX_PERFORMANCE_HOURS, + normalizeFilters, + normalizePerformanceHours, + toModelQuery, + toQuery, +} from './filters' + +describe('production performance filters', () => { + test('defaults the initial view and invalid fallback to the latest hour', () => { + assert.equal(DEFAULT_FILTERS.hours, '1') + assert.equal(toQuery(DEFAULT_FILTERS).hours, DEFAULT_PERFORMANCE_HOURS) + assert.equal( + toQuery({ ...DEFAULT_FILTERS, hours: '' }).hours, + DEFAULT_PERFORMANCE_HOURS + ) + }) + + test('accepts a custom integer hour range', () => { + assert.equal(toQuery({ ...DEFAULT_FILTERS, hours: '37' }).hours, 37) + assert.equal( + toModelQuery({ ...DEFAULT_MODEL_FILTERS, hours: '37' }).hours, + 37 + ) + }) + + test('normalizes custom hours to the supported query window', () => { + assert.equal(normalizePerformanceHours('0'), DEFAULT_PERFORMANCE_HOURS) + assert.equal(normalizePerformanceHours('-5'), DEFAULT_PERFORMANCE_HOURS) + assert.equal(normalizePerformanceHours('12.8'), 12) + assert.equal(normalizePerformanceHours('999'), MAX_PERFORMANCE_HOURS) + assert.equal( + normalizeFilters({ ...DEFAULT_FILTERS, hours: '999' }).hours, + String(MAX_PERFORMANCE_HOURS) + ) + }) + + test('accepts only positive whole channel IDs', () => { + for (const channelId of ['', '0', '-3', '12abc', '12.5', '1e2']) { + assert.equal( + toQuery({ ...DEFAULT_FILTERS, channelId }).channelId, + undefined + ) + } + assert.equal(toQuery({ ...DEFAULT_FILTERS, channelId: '12' }).channelId, 12) + }) + + test('trims optional text filters and drops whitespace-only values', () => { + assert.deepEqual( + toQuery({ + ...DEFAULT_FILTERS, + model: ' alpha ', + group: ' prod ', + }), + { + hours: DEFAULT_PERFORMANCE_HOURS, + channelId: undefined, + model: 'alpha', + group: 'prod', + } + ) + assert.equal(toQuery({ ...DEFAULT_FILTERS, model: ' ' }).model, undefined) + }) +}) diff --git a/web/default/src/features/smart-ops/lib/filters.ts b/web/default/src/features/smart-ops/lib/filters.ts new file mode 100644 index 00000000..3483b4a8 --- /dev/null +++ b/web/default/src/features/smart-ops/lib/filters.ts @@ -0,0 +1,91 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import type { ChannelPerformanceQuery, ModelPerformanceQuery } from '../types' + +export type FilterDraft = { + hours: string + channelId: string + model: string + group: string +} + +export const DEFAULT_PERFORMANCE_HOURS = 1 +export const MAX_PERFORMANCE_HOURS = 168 +export const MAX_PERFORMANCE_LIMIT = 200 + +export const DEFAULT_FILTERS: FilterDraft = { + hours: String(DEFAULT_PERFORMANCE_HOURS), + channelId: '', + model: '', + group: '', +} + +export function normalizePerformanceHours(value: string): number { + const parsed = Number(value) + if (!Number.isFinite(parsed)) return DEFAULT_PERFORMANCE_HOURS + return Math.min( + Math.max(Math.trunc(parsed), DEFAULT_PERFORMANCE_HOURS), + MAX_PERFORMANCE_HOURS + ) +} + +export function normalizeFilters(filters: T): T { + return { + ...filters, + hours: String(normalizePerformanceHours(filters.hours)), + } as T +} + +export function toQuery(filters: FilterDraft): ChannelPerformanceQuery { + const channelId = parsePositiveChannelId(filters.channelId) + return { + hours: normalizePerformanceHours(filters.hours), + channelId, + model: filters.model.trim() || undefined, + group: filters.group.trim() || undefined, + } +} + +function parsePositiveChannelId(value: string): number | undefined { + const normalized = value.trim() + if (!/^\+?\d+$/.test(normalized)) return undefined + const channelId = Number(normalized) + if (!Number.isSafeInteger(channelId) || channelId <= 0) return undefined + return channelId +} + +export type ModelFilterDraft = { + hours: string + model: string + group: string +} + +export const DEFAULT_MODEL_FILTERS: ModelFilterDraft = { + hours: String(DEFAULT_PERFORMANCE_HOURS), + model: '', + group: '', +} + +export function toModelQuery(filters: ModelFilterDraft): ModelPerformanceQuery { + return { + hours: normalizePerformanceHours(filters.hours), + model: filters.model.trim() || undefined, + group: filters.group.trim() || undefined, + } +} diff --git a/web/default/src/features/smart-ops/lib/format.test.ts b/web/default/src/features/smart-ops/lib/format.test.ts new file mode 100644 index 00000000..0fd45386 --- /dev/null +++ b/web/default/src/features/smart-ops/lib/format.test.ts @@ -0,0 +1,33 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import assert from 'node:assert/strict' +import { describe, test } from 'node:test' +import { formatLegacyLatency } from './format' + +describe('formatLegacyLatency', () => { + test('keeps a recorded zero-second average distinct from missing data', () => { + assert.equal(formatLegacyLatency(0), '0ms') + assert.equal(formatLegacyLatency(null), '—') + }) + + test('uses the shared latency format for positive values', () => { + assert.equal(formatLegacyLatency(500), '500ms') + assert.equal(formatLegacyLatency(1_500), '1.50s') + }) +}) diff --git a/web/default/src/features/smart-ops/lib/format.ts b/web/default/src/features/smart-ops/lib/format.ts new file mode 100644 index 00000000..d4284678 --- /dev/null +++ b/web/default/src/features/smart-ops/lib/format.ts @@ -0,0 +1,33 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import { formatLatency } from '@/features/performance-metrics/lib/format' + +export function formatLegacyLatency( + milliseconds: number | null | undefined +): string { + if ( + milliseconds == null || + !Number.isFinite(milliseconds) || + milliseconds < 0 + ) { + return '—' + } + if (milliseconds === 0) return '0ms' + return formatLatency(milliseconds) +} diff --git a/web/default/src/features/smart-ops/lib/model-sort.test.ts b/web/default/src/features/smart-ops/lib/model-sort.test.ts new file mode 100644 index 00000000..88bc7283 --- /dev/null +++ b/web/default/src/features/smart-ops/lib/model-sort.test.ts @@ -0,0 +1,142 @@ +import assert from 'node:assert/strict' +import { describe, test } from 'node:test' +import type { ModelPerformanceItem } from '../types' +import { + sortModelPerformanceItems, + type ModelPerformanceSortKey, +} from './model-sort' + +function performanceItem( + modelName: string, + values: Partial +): ModelPerformanceItem { + return { + model_name: modelName, + channel_count: 1, + observed_count: 0, + consume_log_count: 0, + error_log_count: 0, + consumed_quota: 0, + retry_log_count: 0, + latency_sample_count: 0, + observed_success_rate: null, + avg_logged_latency_ms: null, + avg_tps: null, + last_observed_at: 1, + quality_flags: [], + ...values, + } +} + +const rows = [ + performanceItem('Balanced model', { + channel_count: 2, + observed_count: 20, + error_log_count: 2, + consumed_quota: 800, + observed_success_rate: 90, + avg_logged_latency_ms: 1000, + avg_tps: 60, + retry_log_count: 1, + }), + performanceItem('Problem model', { + channel_count: 1, + observed_count: 5, + error_log_count: 8, + consumed_quota: 200, + observed_success_rate: 40, + avg_logged_latency_ms: 4500, + avg_tps: 10, + retry_log_count: 4, + }), + performanceItem('Unknown metrics', { + observed_count: 10, + error_log_count: 1, + consumed_quota: 0, + retry_log_count: null, + avg_tps: null, + }), +] + +describe('sortModelPerformanceItems', () => { + const cases: Array<{ + key: ModelPerformanceSortKey + asc: string[] + desc: string[] + }> = [ + { + key: 'model_name', + asc: ['Balanced model', 'Problem model', 'Unknown metrics'], + desc: ['Unknown metrics', 'Problem model', 'Balanced model'], + }, + { + key: 'channel_count', + asc: ['Problem model', 'Unknown metrics', 'Balanced model'], + desc: ['Balanced model', 'Problem model', 'Unknown metrics'], + }, + { + key: 'observed_count', + asc: ['Problem model', 'Unknown metrics', 'Balanced model'], + desc: ['Balanced model', 'Unknown metrics', 'Problem model'], + }, + { + key: 'error_log_count', + asc: ['Unknown metrics', 'Balanced model', 'Problem model'], + desc: ['Problem model', 'Balanced model', 'Unknown metrics'], + }, + { + key: 'consumed_quota', + asc: ['Unknown metrics', 'Problem model', 'Balanced model'], + desc: ['Balanced model', 'Problem model', 'Unknown metrics'], + }, + { + key: 'observed_success_rate', + asc: ['Problem model', 'Balanced model', 'Unknown metrics'], + desc: ['Balanced model', 'Problem model', 'Unknown metrics'], + }, + { + key: 'avg_logged_latency_ms', + asc: ['Balanced model', 'Problem model', 'Unknown metrics'], + desc: ['Problem model', 'Balanced model', 'Unknown metrics'], + }, + { + key: 'avg_tps', + asc: ['Problem model', 'Balanced model', 'Unknown metrics'], + desc: ['Balanced model', 'Problem model', 'Unknown metrics'], + }, + { + key: 'retry_log_count', + asc: ['Balanced model', 'Problem model', 'Unknown metrics'], + desc: ['Problem model', 'Balanced model', 'Unknown metrics'], + }, + ] + + for (const { key, asc, desc } of cases) { + test(`sorts ${key} in both directions and keeps missing values last`, () => { + assert.deepEqual( + sortModelPerformanceItems(rows, { key, direction: 'asc' }).map( + (item) => item.model_name + ), + asc + ) + assert.deepEqual( + sortModelPerformanceItems(rows, { key, direction: 'desc' }).map( + (item) => item.model_name + ), + desc + ) + }) + } + + test('does not mutate the API result order', () => { + const originalOrder = rows.map((item) => item.model_name) + sortModelPerformanceItems(rows, { + key: 'error_log_count', + direction: 'desc', + }) + assert.deepEqual( + rows.map((item) => item.model_name), + originalOrder + ) + }) +}) diff --git a/web/default/src/features/smart-ops/lib/model-sort.ts b/web/default/src/features/smart-ops/lib/model-sort.ts new file mode 100644 index 00000000..5ec956aa --- /dev/null +++ b/web/default/src/features/smart-ops/lib/model-sort.ts @@ -0,0 +1,55 @@ +import type { ModelPerformanceItem } from '../types' + +export type ModelPerformanceSortKey = + | 'model_name' + | 'channel_count' + | 'observed_count' + | 'error_log_count' + | 'consumed_quota' + | 'observed_success_rate' + | 'avg_logged_latency_ms' + | 'avg_tps' + | 'retry_log_count' + +export type ModelPerformanceSortState = { + key: ModelPerformanceSortKey + direction: 'asc' | 'desc' +} | null + +function sortableValue( + item: ModelPerformanceItem, + key: ModelPerformanceSortKey +): number | string | null { + const value = item[key] + if (key === 'model_name') return typeof value === 'string' ? value : null + return typeof value === 'number' && Number.isFinite(value) ? value : null +} + +export function sortModelPerformanceItems( + items: readonly ModelPerformanceItem[], + sortState: ModelPerformanceSortState, + locale?: string +): readonly ModelPerformanceItem[] { + if (!sortState) return items + + return items + .map((item, originalIndex) => ({ item, originalIndex })) + .sort((left, right) => { + const leftValue = sortableValue(left.item, sortState.key) + const rightValue = sortableValue(right.item, sortState.key) + + if (leftValue == null && rightValue == null) { + return left.originalIndex - right.originalIndex + } + if (leftValue == null) return 1 + if (rightValue == null) return -1 + + const comparison = + typeof leftValue === 'string' && typeof rightValue === 'string' + ? leftValue.localeCompare(rightValue, locale) + : Number(leftValue) - Number(rightValue) + if (comparison === 0) return left.originalIndex - right.originalIndex + return sortState.direction === 'asc' ? comparison : -comparison + }) + .map(({ item }) => item) +} diff --git a/web/default/src/features/smart-ops/lib/sort-direction-icon.tsx b/web/default/src/features/smart-ops/lib/sort-direction-icon.tsx new file mode 100644 index 00000000..3c11be2e --- /dev/null +++ b/web/default/src/features/smart-ops/lib/sort-direction-icon.tsx @@ -0,0 +1,36 @@ +/* +Copyright (C) 2023-2026 MAX-API-Next + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . + +For commercial licensing, please contact https://github.com/MAX-API-Next/MAX-API/issues +*/ +import { ArrowDown, ArrowUp, ChevronsUpDown } from 'lucide-react' +import type { SortDirection } from './sort-direction' + +type SortDirectionIconProps = { + direction: SortDirection +} + +export function SortDirectionIcon( + props: SortDirectionIconProps +): React.ReactElement { + if (props.direction === 'desc') { + return
{/* Admin Actions */} - {isAdmin && record.status === 'pending' && ( + {isAdmin && isCompletableTopupStatus(record.status) && (