Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/code-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
dotnet-version: '10.x'

- name: Run focused tests
run: dotnet test src/HuaJiBot.NET.UnitTest --filter 'FullyQualifiedName~DailySummaryTest|FullyQualifiedName~AsciiArtToolsTest|FullyQualifiedName~WebDataServiceTest'
run: dotnet test src/HuaJiBot.NET.UnitTest --filter 'FullyQualifiedName~DailySummaryTest|FullyQualifiedName~AsciiArtToolsTest|FullyQualifiedName~WebDataServiceTest|FullyQualifiedName~VisionInputTest'

format-check:
runs-on: ubuntu-latest
Expand Down
7 changes: 7 additions & 0 deletions deploy/aichat-multimodal/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Build context contains linux-x64 CLI publish output and the Release AIChat DLL.
# Overlay the current production image to preserve its other plugins and MCP services.
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
COPY cli/ /app/bin/
COPY --chmod=755 cli/HuaJiBot.NET.CLI /app/bin/HuaJiBot.NET.CLI
COPY aichat/HuaJiBot.NET.Plugin.AIChat.dll /app/plugins/HuaJiBot.NET.Plugin.AIChat.dll
66 changes: 66 additions & 0 deletions deploy/aichat-multimodal/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# AIChat image input

The OneBot adapter exposes image attachments through `GroupMessageEventArgs.ImageUrls`.
Typed URLs remain text. AIChat downloads attachments and sends validated image bytes as
Chat Completions image blocks. This first implementation covers OneBot/QQ; other adapters
can populate the same optional event field. PR #65 and its KOOK adapter remain separate.

Under `Plugins.AIChat`, set `SupportsVision: true` only for a model with verified image
input support. `agnes-3.0-flash` and `agnes-2.5-flash` document image understanding;
`agnes-image-2.5-flash` is an image generation endpoint, not a chat model.
Defaults: `MaxImagesPerRequest: 4`, `MaxImageBytes: 8388608` (per image).
Requests include referenced historical images in the count. Old history remains readable.
Visual requests use reconstructed reply history and short-lived agent sessions, so image
bytes do not accumulate in the per-group session cache.

Users can send `@bot <question> + image`, `@bot + image` (default image description),
or reply to a bot response with a follow-up question. Images from earlier AI exchanges
are restored from history. Replying to an arbitrary image that never reached AIChat is
not supported. Expired image URLs require resending the image.

Only public HTTP(S) attachment addresses are downloaded. Redirects and private-network
destinations are rejected. Download time is capped at 20 seconds and the complete request
at 90 seconds. PNG, JPEG, WebP and GIF are supported. Downloads are held in memory;
history stores source URLs, not Base64. AI request logs report counts only, and SDK
content logging is disabled when vision is enabled. An unsupported model, failed image
download, invalid format or exceeded limit produces an explicit user-facing message.

## Validation

```sh
dotnet test src/HuaJiBot.NET.UnitTest --filter FullyQualifiedName~VisionInputTest
```

Tests cover OneBot parsing, real image content and OpenAI-compatible serialization,
text-only behavior, capability gating, historical image limits, legacy history, invalid
URLs, network address checks, download failure and content validation. Real acceptance
requires a non-bot account to send a picture and mention in the configured QQ group,
then confirm the response describes details visible only in the picture and that a
reply-based follow-up retains the image context.

## Production overlay deployment

Following Issue #38, stop the bot before changing configuration. Back up `compose.yaml`,
`config.json` and plugin data; tag the current image for rollback. Do not restart other
Compose services. Build a Linux CLI and the Release AIChat plugin from the same commit:

```sh
dotnet publish src/HuaJiBot.NET.CLI -c Release -f net10.0 -r linux-x64 \
--no-self-contained -p:PublishSingleFile=true -o /tmp/huajibot-vision/cli
dotnet build src/HuaJiBot.NET.Plugin.AIChat -c Release -o /tmp/huajibot-vision/aichat
docker build --build-arg BASE_IMAGE=<current-production-image> \
-f deploy/aichat-multimodal/Dockerfile -t <new-image> /tmp/huajibot-vision
```

The overlay assumes production already includes this branch's AIChat dependencies
(Microsoft.Agents.AI 1.1.0 and Microsoft.Extensions.AI.OpenAI 10.4.1). Verify versions
before using it; if dependencies differ, build the full image or update dependencies
explicitly. It preserves the current PushPanel and other plugin binaries.

With the bot stopped, enable vision, change only its Compose image and start:
`docker compose up -d --no-deps huaji-bot-dotnet`. Verify startup, OneBot connection,
plugin loading, real image response, historical follow-up, text-only response and
existing panel behavior. Compare configuration against backup excluding the new vision
fields. Roll back by restoring the backed-up Compose file and config while stopped,
then start only the bot. Preserve the pre-deployment plugin data backup for recovery;
do not overwrite new chat history unless recovery specifically requires it.
3 changes: 3 additions & 0 deletions src/HuaJiBot.NET.Adapter.OneBot/CommandReader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ IEnumerable<ReaderEntity> Parse()
case ReplyMessageEntity { MessageId: var id }:
yield return new ReaderReply(new(messageId: id));
break;
case ImageMessageEntity:
// Images are provided separately through GroupMessageEventArgs.ImageUrls.
break;
default:
service.LogDebug($"未转换消息类型:{element.GetType().Name}");
break;
Expand Down
8 changes: 8 additions & 0 deletions src/HuaJiBot.NET.Adapter.OneBot/OneBotMessageHandler.cs
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ public async Task ProcessMessageAsync(JObject json)
_ => MemberType.Unknown,
},
TextMessageLazy = new(() => ToPlainText(message)),
ImageUrls = ExtractImageUrls(message),
}
);
}
Expand All @@ -233,6 +234,13 @@ public async Task ProcessMessageAsync(JObject json)
#endif
}

internal static string[] ExtractImageUrls(IEnumerable<MessageEntity> messages) =>
messages.OfType<ImageMessageEntity>()
.Select(image => string.IsNullOrWhiteSpace(image.Url) ? image.File : image.Url)
.Where(url => !string.IsNullOrWhiteSpace(url))
.Distinct(StringComparer.Ordinal)
.ToArray();

private static string ToPlainText(IEnumerable<MessageEntity> message) =>
string.Concat(
message.Select(x =>
Expand Down
4 changes: 4 additions & 0 deletions src/HuaJiBot.NET.Plugin.AIChat/Config/PluginConfig.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ public class PluginConfig : ConfigBase
{
public string SystemPrompt = "你是一个有用的AI助手";
public ModelConfig Model = new();
/// <summary>Enable only for models that accept image content.</summary>
public bool SupportsVision = false;
public int MaxImagesPerRequest = 4;
public int MaxImageBytes = 8 * 1024 * 1024;

/// <summary>
/// Groups where the bot answers. Empty answers no group.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
</ItemGroup>
<ItemGroup>
<EmbeddedResource Include="Fonts\ANSI Shadow.flf" />
<InternalsVisibleTo Include="HuaJiBot.NET.UnitTest" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\HuaJiBot.NET\HuaJiBot.NET.csproj" />
Expand Down
52 changes: 30 additions & 22 deletions src/HuaJiBot.NET.Plugin.AIChat/PluginMain.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
using HuaJiBot.NET.Plugin.AIChat.Service;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
using Newtonsoft.Json;
using System.Collections.Concurrent;

namespace HuaJiBot.NET.Plugin.AIChat;
Expand All @@ -18,7 +17,9 @@ private sealed record GroupSession(AgentSession Session, DateTimeOffset LastUsed
private static readonly TimeSpan SessionIdleTimeout = TimeSpan.FromHours(1);
private static readonly TimeSpan LlmTimeout = TimeSpan.FromSeconds(90);
private readonly ConcurrentDictionary<string, GroupSession> _sessions = new();
private AgentConnector Connector => AgentConnector.Create(Service, Config.Model);
private AgentConnector Connector => AgentConnector.Create(Service,
Config.SupportsVision ? Config.Model with { Logging = false } : Config.Model);
private readonly VisionInput _visionInput = new();

private AIFunction[]? _tools;

Expand Down Expand Up @@ -103,6 +104,10 @@ Events.GroupMessageEventArgs e
Warn($"群组 {e.GroupId} 的 AI 请求超过 {LlmTimeout.TotalSeconds:0} 秒");
await e.Reply("模型响应超时,请稍后重试。");
}
catch (VisionInputException exception)
{
await e.Reply(exception.Message);
}
finally
{
_busyGroups.TryRemove(e.GroupId, out _);
Expand All @@ -116,21 +121,21 @@ private async Task InvokeLlmMessageCore(
CancellationToken cancellationToken
)
{
Info(
"调用AI消息\n\t"
+ JsonConvert
.SerializeObject(
messages,
new JsonSerializerSettings
{
Formatting = Formatting.Indented,
NullValueHandling = NullValueHandling.Ignore,
}
)
.Replace("\n", "\n\t")
);
await _visionInput.PrepareAsync(messages, Config.SupportsVision, Config.MaxImagesPerRequest,
Config.MaxImageBytes, cancellationToken);
Info($"调用AI:{messages.Count} 条消息,{messages.Sum(message => message.Contents.OfType<DataContent>().Count())} 张图片");
var connector = Connector;
var session = await GetSessionAsync(connector, e.GroupId, cancellationToken);
AgentSession session;
if (messages.Any(message => message.Contents.OfType<DataContent>().Any()))
{
// Reply context is already reconstructed above. Do not retain image bytes across turns.
_sessions.TryRemove(e.GroupId, out _);
session = await connector.CreateSessionAsync(cancellationToken);
}
else
{
session = await GetSessionAsync(connector, e.GroupId, cancellationToken);
}
var agent = connector.CreateAIAgentWithOptions(
BuildSystemPrompt(
systemPrompt,
Expand Down Expand Up @@ -200,7 +205,8 @@ private async Task OnGroupMessageReceivedAsync(Events.GroupMessageEventArgs e)
{
if (!Service.AllRobots.Contains(atId))
return; //仅处理At机器人
if (!reader.Input(out var restText, true) || string.IsNullOrWhiteSpace(restText))
_ = reader.Input(out var restText, true);
if (string.IsNullOrWhiteSpace(restText) && e.ImageUrls.Length == 0)
{
try
{
Expand All @@ -221,6 +227,7 @@ private async Task OnGroupMessageReceivedAsync(Events.GroupMessageEventArgs e)
new GroupMessage
{
Content = restText,
ImageUrls = e.ImageUrls,
GroupId = e.GroupId,
MessageId = e.MessageId,
SenderId = e.SenderId,
Expand All @@ -229,7 +236,7 @@ private async Task OnGroupMessageReceivedAsync(Events.GroupMessageEventArgs e)
ReplyToMessageId = null,
}
);
if (AsciiArtTools.TryRenderCommand(restText, out var asciiArt))
if (e.ImageUrls.Length == 0 && AsciiArtTools.TryRenderCommand(restText!, out var asciiArt))
{
// Send raw text. Markdown conversion changes FIGlet spacing and line breaks.
foreach (var msgId in await e.Reply(asciiArt))
Expand All @@ -252,7 +259,7 @@ private async Task OnGroupMessageReceivedAsync(Events.GroupMessageEventArgs e)
//调用LLM回复
await InvokeLlmMessage(
Config.SystemPrompt,
[new ChatMessage(ChatRole.User, restText)],
[VisionInput.CreateMessage(restText, e.ImageUrls)],
e
);
}
Expand Down Expand Up @@ -295,7 +302,7 @@ void PrependMessage(GroupMessage message)
void FetchReply(string replyId)
{
var replyMessage = _history.GetMessage(replyId);
if (replyMessage is not null)
if (replyMessage is not null && replyMessage.GroupId == e.GroupId)
{ //获取到被回复的消息
PrependMessage(replyMessage);
}
Expand Down Expand Up @@ -335,16 +342,17 @@ void FetchReply(string replyId)
prompts.Add(
message.IsBot
? new ChatMessage(ChatRole.Assistant, message.Content)
: new ChatMessage(ChatRole.User, message.Content)
: VisionInput.CreateMessage(message.Content, message.ImageUrls)
);
}
prompts.Add(new ChatMessage(ChatRole.User, text));
prompts.Add(VisionInput.CreateMessage(text, e.ImageUrls));

//收到回复消息记录
_history.StoreMessage(
new GroupMessage
{
Content = text,
ImageUrls = e.ImageUrls,
GroupId = e.GroupId,
MessageId = e.MessageId,
SenderId = e.SenderId,
Expand Down
Loading
Loading