The Model Context Protocol (MCP) defines a client-server protocol for AI agents to access tools and data. An MCP client (the agent) connects to an MCP server (a process that wraps a tool or data source) and invokes tools through a JSON-RPC interface. The transport between client and server can be stdio (same process) or HTTP+SSE (network).
+For a closer look at how webhooks and SSE streaming deliver results for long-running research jobs — and where the reachable-endpoint requirement gets fragile — see how webhooks and SSE streaming work for long-running jobs.
+When people refer to "MCP tunnels," they are usually talking about the transport layer between an MCP client and server over a network — a persistent or long-lived HTTP connection (often using Server-Sent Events) through which tool calls and results flow. Some implementations wrap this in WebSocket connections for bidirectional streaming. The key property is that MCP tunnels connect an agent to its tools — databases, APIs, file systems, search engines — not to other agents.
MCP tunnels give you:
diff --git a/src/pages/learn/nats-vs-grpc-agent-messaging.astro b/src/pages/learn/nats-vs-grpc-agent-messaging.astro index 2e6dddac..5ac53bd1 100644 --- a/src/pages/learn/nats-vs-grpc-agent-messaging.astro +++ b/src/pages/learn/nats-vs-grpc-agent-messaging.astro @@ -39,6 +39,8 @@ const bodyContent = `You are building agents that need to talk to each other.
gRPC organizes communication around service definitions in Protocol Buffers. You define a service with methods — some unary (one request, one response), some server-streaming (one request, stream of responses), some bidirectional.
+Server-streaming is how gRPC handles long-running work: the client opens one call and receives a stream of responses. The same shape appears over plain HTTP as Server-Sent Events, with webhooks covering the completion case — see how webhooks and SSE streaming work for long-running research jobs for how those patterns behave under parallel task fan-out.
+The contract-first approach is a practical advantage for teams: the .proto file is a single source of truth for the API surface. Code generation produces client and server stubs in twelve-plus languages, so you cannot accidentally send the wrong type. For structured agent interactions — submit a task, get a result — gRPC is ergonomic.
The limit is that gRPC assumes you know who you are calling. Service discovery, load balancing, and failover are not part of the framework — they come from infrastructure (DNS SRV, Consul, Kubernetes Services). If you have 200 agents and any one of them can call any other, you need a way for them to discover each other's addresses and health status.
diff --git a/src/pages/learn/webhooks-sse-streaming-long-running-jobs.astro b/src/pages/learn/webhooks-sse-streaming-long-running-jobs.astro new file mode 100644 index 00000000..89af868e --- /dev/null +++ b/src/pages/learn/webhooks-sse-streaming-long-running-jobs.astro @@ -0,0 +1,165 @@ +--- +import BlogLayout from "../../layouts/BlogLayout.astro"; + +const bodyContent = ` +++TL;DR:
++
+- Webhooks deliver the finished result: the job runs asynchronously, then POSTs the result to a callback URL you registered when you started it. They are ideal for completion events, retries, and fan-in from parallel subtasks.
+- SSE (Server-Sent Events) streams progress: the server holds an HTTP connection open and pushes progress events and partial results down it as they appear. One-way, server to client, built for live updates.
+- Parallel research jobs combine them: a coordinator fans out subtasks, each worker streams progress over SSE, and results fan back in through a shared callback — or per-task callbacks with correlation IDs.
+- Both approaches assume the receiving end is publicly reachable. When the receiver is an AI agent behind NAT, that assumption is the fragile part — a persistent tunnel with a stable address removes it.
+
If you are building a research system where jobs run for minutes to hours — deep research, document analysis, codebase investigation — you will eventually ask how parallel task webhooks and SSE streaming work for long-running research jobs. The short answer: webhooks deliver the finished result to a callback URL, SSE streams progress over a connection the server keeps open, and production systems usually combine both. The longer answer is about fan-out, retries, idempotency, and one requirement both approaches quietly share: something has to be reachable at the end of the call.
+ +This post walks through the mechanics of each pattern, how they behave under parallel task fan-out, where they break in agent-based systems, and what a persistent delivery layer changes.
+ +A webhook is a callback over HTTP. The flow looks like this: your client submits a job to a worker or job API and gets back a job ID. The worker runs the job asynchronously — this is the part that takes minutes or hours. When the job finishes, the worker makes an outbound POST to the callback URL you supplied at submit time, with the result in the body. Your side acknowledges the POST, and the job is complete.
+ +Three details make webhooks work in practice:
+When a research job fans out into parallel subtasks, you have two designs. In per-task callbacks, each subtask POSTs its own result to the callback URL. The receiver correlates results by subtask ID and waits until the expected set arrives. This is simple but chatty, and results arrive out of order — ordering must be reconstructed by the receiver.
+ +In fan-in, subtasks report to a coordinator inside the job, and the coordinator POSTs one aggregate callback when the whole job completes. Fewer endpoints, deterministic ordering, one place to implement retries and idempotency. Most production research pipelines use fan-in with a per-subtask progress channel on top — which is where SSE enters.
+ +Server-Sent Events is the standard way to stream one-way updates over plain HTTP. The client opens a GET request with Accept: text/event-stream. The server keeps the connection open and writes events as frames: event: names the event type, data: carries the payload, id: marks the position in the stream, and comment lines (starting with a colon) act as heartbeats to keep intermediaries from closing idle connections.
Two properties make SSE attractive for research workloads:
+Last-Event-ID, letting the server resume from where the client left off.SSE is one-way by design: server to client. The client cannot push messages back over the same connection; it uses ordinary requests for that. And SSE is still HTTP — it runs through reverse proxies and load balancers, which means those layers must be configured for long-lived connections: buffering off, read and idle timeouts raised, and connection limits accounted for. Each open SSE connection occupies a socket on both ends for the entire job.
+ +In a parallel research job, SSE typically carries progress from each worker to the coordinator, while the final result arrives through the webhook path. The two patterns complement each other: SSE for the live view, webhook for the authoritative completion signal.
+ +| Mechanism | Best for | What it assumes |
|---|---|---|
| Polling | Short jobs, simple clients, no push infrastructure | The client can keep asking; wasted requests are acceptable |
| Webhook | Completion of long jobs, fan-in from parallel subtasks | The callback receiver is publicly reachable and the URL survives the job |
| SSE | Live progress, partial results, dashboards | The connection stays open; proxies and load balancers cooperate |
| Hybrid (SSE + webhook) | Production research pipelines | Both of the above, plus a coordinator to join the two paths |
The table makes the tradeoff visible: the more live feedback you want, the more infrastructure you need to keep connections and endpoints healthy. For long-running research jobs, the hybrid is the common answer — and its failure modes are almost never in the transport choice.
+ +Every pattern above shares one assumption: the receiving end is reachable at a stable URL. That assumption fails in specific, predictable ways:
+ +None of this is a criticism of webhooks or SSE — they are the right tools inside a trusted boundary with reachable endpoints. The problems start exactly where agent-based systems live: distributed, behind NAT, and restarted without ceremony.
+ +If the fragile part is public reachability, the fix is a delivery layer that does not require it. That is the problem an agent-native overlay network is built to solve. Pilot Protocol gives every agent a permanent virtual address that survives restarts, IP changes, and moves across clouds. Traffic travels over encrypted UDP tunnels — X25519 key exchange with AES-GCM — and NAT traversal (STUN with hole-punching and a relay fallback) means agents behind NAT are reachable without port forwarding or a public host.
+ +The research pipeline looks different on this layer. The coordinator agent registers its stable address. Workers connect to it — outbound, like a webhook client — and the tunnel stays up for the life of the job. Progress streams through the tunnel the way SSE events would, and the final result lands on the coordinator's address, which does not change when the process restarts. Trust is explicit: agents approve a mutual handshake before any traffic flows, so there is no open callback endpoint for an attacker to discover.
+ +Discovery is part of the same layer. A rendezvous registry lets agents find each other by name or tag, so the coordinator does not need to hand out URLs — workers resolve the agent they are working for. For builders, the Pilot app store adds installable capability apps — grounded search, web-to-markdown, runtime security — that run locally on the daemon, discovered and installed with a single command.
+ +For a deeper look at when replacing webhooks with persistent tunnels makes sense, see network tunnels for AI agent communication. For how this layer compares with the HTTP+SSE transport that MCP tunnels use, see MCP tunnels vs VPN for AI agents.
+ +Get started with one command:
+ +curl -fsSL https://pilotprotocol.network/install.sh | shA webhook is a one-way HTTP POST sent by the server to a callback URL when something completes — one request, one response. SSE is a long-lived HTTP connection over which the server pushes many events over time. Webhooks answer "is it done?", SSE answers "what is happening right now?"
+ +Yes, in two shapes: per-task callbacks (each subtask POSTs its own result, correlated by subtask ID) or fan-in (subtasks report to a coordinator that POSTs one aggregate callback when the whole job finishes). Fan-in is simpler to make idempotent and ordered.
+ +SSE is designed for live progress and partial results. Delivering the authoritative final result over a long-lived connection is risky — proxies and load balancers can close idle or long-lived connections, and a dropped connection after an hour of work is a bad place to lose the result. Most systems stream progress over SSE and deliver the final result over a webhook or a dedicated fetch.
+ +A webhook delivery is an inbound connection: the worker initiates a connection to your callback URL. Behind NAT, inbound connections cannot reach the agent unless port forwarding or a reverse proxy is configured. An agent-native overlay with NAT traversal removes this requirement — the tunnel is established from the agent's side and inbound traffic arrives through it.
+ +No. Webhooks and SSE are delivery patterns — they remain the right choice inside a trusted boundary with reachable endpoints. Pilot Protocol replaces the fragile assumption under them: it gives each agent a stable virtual address and an encrypted tunnel that works across NAT and survives restarts, so results can be delivered to the agent itself instead of to a public URL that may not exist anymore.
+ +A fan-in callback is a single webhook posted after a parallel job completes: subtasks report their results to a coordinator inside the job, the coordinator aggregates them, and one POST carries the combined result to the callback URL. It reduces endpoint churn, makes ordering deterministic, and centralizes retry and idempotency logic.
+`; + +const faqItems = [ + { + question: "What is the difference between a webhook and SSE?", + answer: "A webhook is a one-way HTTP POST sent by the server to a callback URL when something completes — one request, one response. SSE is a long-lived HTTP connection over which the server pushes many events over time. Webhooks answer \"is it done?\", SSE answers \"what is happening right now?\"", + }, + { + question: "Can webhooks handle parallel tasks?", + answer: "Yes, in two shapes: per-task callbacks (each subtask POSTs its own result, correlated by subtask ID) or fan-in (subtasks report to a coordinator that POSTs one aggregate callback when the whole job finishes). Fan-in is simpler to make idempotent and ordered.", + }, + { + question: "Is SSE good for delivering the final result of a long job?", + answer: "SSE is designed for live progress and partial results. Delivering the authoritative final result over a long-lived connection is risky — proxies and load balancers can close long-lived connections, and a dropped connection after an hour of work is a bad place to lose the result. Most systems stream progress over SSE and deliver the final result over a webhook or a dedicated fetch.", + }, + { + question: "Why do webhooks fail for agents behind NAT?", + answer: "A webhook delivery is an inbound connection: the worker initiates a connection to your callback URL. Behind NAT, inbound connections cannot reach the agent unless port forwarding or a reverse proxy is configured. An agent-native overlay with NAT traversal removes this requirement — the tunnel is established from the agent's side and inbound traffic arrives through it.", + }, + { + question: "Does Pilot Protocol replace webhooks and SSE?", + answer: "No. Webhooks and SSE are delivery patterns — they remain the right choice inside a trusted boundary with reachable endpoints. Pilot Protocol replaces the fragile assumption under them: it gives each agent a stable virtual address and an encrypted tunnel that works across NAT and survives restarts, so results can be delivered to the agent itself instead of to a public URL that may not exist anymore.", + }, + { + question: "What is a fan-in callback?", + answer: "A fan-in callback is a single webhook posted after a parallel job completes: subtasks report their results to a coordinator inside the job, the coordinator aggregates them, and one POST carries the combined result to the callback URL. It reduces endpoint churn, makes ordering deterministic, and centralizes retry and idempotency logic.", + }, +]; +--- +