Skip to content
Open

Steps #753

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .changeset/d8a-purchase-example.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
'@walkeros/web-destination-d8a': patch
---

The purchase example's item condition works as published, so the example no
longer throws when a tool evaluates it from the package metadata. The
`send_page_view` description says d8a sends a page view when the tracker starts
unless it is set to false.
9 changes: 9 additions & 0 deletions .changeset/fingerprint-ip-not-an-address.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'@walkeros/server-transformer-fingerprint': patch
'@walkeros/cli': patch
---

An `ip` input that is not one address, such as a forwarded chain, is now
reported as an error and left out of the hash, instead of a warning that said it
was empty. The event is still delivered. The complete example flow reads the
visitor IP from a header the proxy sets.
8 changes: 8 additions & 0 deletions .changeset/linkedin-conversion-description.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
'@walkeros/web-destination-linkedin': patch
---

The `conversion` mapping description now says what happens when the conversion
resolves to no id: no `lintrk` call is made and the push fails with
`missing-input`. It no longer says such events are ignored, and it names the
server LinkedIn destination as the deduplication partner of `eventId`.
8 changes: 8 additions & 0 deletions .changeset/optimizely-user-id-description.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
'@walkeros/web-destination-optimizely': patch
---

The `userId` description shows a working fallback, `["user.id", "user.device"]`,
instead of a `{ key, value }` form that sent the literal text `user.device` as
the user id of every anonymous visitor. The `attributes` description says the
attributes apply when the user context is created.
10 changes: 10 additions & 0 deletions .changeset/pinterest-identify-examples.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
'@walkeros/web-destination-pinterest': patch
---

The `identify` description matches the destination: init calls
`pintrk('load', tagId)` without an identity, which is sent with
`pintrk('set', ...)` when it changes, and the tag takes a plain or hashed email.
The checkout example's condition works as published, and the commerce examples
send product fields inside `line_items`, with `product_name` taken from the
product name.
8 changes: 8 additions & 0 deletions .changeset/snowplow-mapping-schema.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
'@walkeros/web-destination-snowplow': patch
---

The mapping schema lists the rule settings `data` (the self-describing event
data) and `struct` (a structured event), which the destination reads, so they
appear in the generated documentation. The `contexts` setting accepts `browser`,
the browser context entity the Snowplow tracker supports.
6 changes: 3 additions & 3 deletions AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,9 @@ apps/
- **No `any`:** Never in production code
- **XP principles:** DRY, KISS, YAGNI, TDD
- **Test first:** Watch it fail before implementing
- **Verify:** Use the verification tiers from `/workspaces/developer/AGENT.md`
rule 11. Default L1 (touched package) per task. L2 at plan completion. Never
L4 unless the plan touched shared infra or the user asked.
- **Verify:** Default L1 per task (`npm run verify:touched -- <pkg>`, the
touched package). L2 at plan completion (`npm run verify:affected`). Never L4
(the full suite) unless the plan touched shared infra or the user asked.

## Creating/Maintaining Skills

Expand Down
3 changes: 2 additions & 1 deletion apps/demos/flows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ production-ready live demo for data ownership and sovereignty.
## Setup

```bash
cd /workspaces/developer/walkerOS/apps/demos/flows
# from the repository root
cd apps/demos/flows

export PROJECT_ID=YOUR_PROJECTID
export DATASET_ID=YOUR_DATASET
Expand Down
3 changes: 0 additions & 3 deletions packages/cli/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -1029,9 +1029,6 @@
and `event.context` remain required.
- Each source self-registers via TypeScript module augmentation of `SourceMap`
in `@walkeros/core`.
- App-side coordination (`/workspaces/developer/app`) is a follow-up plan, not
part of this release. Telemetry from v4 CLI/MCP will not validate against
the existing app schema until that follow-up ships.
- `Mapping.Rule.skip` is renamed to `Mapping.Rule.silent`. Customer flow.json
configs using `skip: true` in mapping rules must rename to `silent: true`.
Hard cut: no legacy alias, the field is gone.
Expand Down
2 changes: 1 addition & 1 deletion packages/cli/examples/flow-complete.json
Original file line number Diff line number Diff line change
Expand Up @@ -1882,7 +1882,7 @@
},
"ingest": {
"map": {
"ip": "ip",
"ip": "headers.x-client-ip",
"path": "path",
"method": "method",
"url": "url",
Expand Down
3 changes: 0 additions & 3 deletions packages/collector/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -711,9 +711,6 @@
and `event.context` remain required.
- Each source self-registers via TypeScript module augmentation of `SourceMap`
in `@walkeros/core`.
- App-side coordination (`/workspaces/developer/app`) is a follow-up plan, not
part of this release. Telemetry from v4 CLI/MCP will not validate against
the existing app schema until that follow-up ships.
- `Mapping.Rule.skip` is renamed to `Mapping.Rule.silent`. Customer flow.json
configs using `skip: true` in mapping rules must rename to `silent: true`.
Hard cut: no legacy alias, the field is gone.
Expand Down
4 changes: 1 addition & 3 deletions packages/collector/src/store-cache-wrapper.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,9 +110,7 @@ export interface WrappedStoreInstance extends Store.Instance {
* number of tiers (negligible for the typical 2-3 tier chain). Each tier owns
* its own `exp` from its own rule, so the staleness bound is per tier.
*
* Write semantics (`set` / `delete`) follow the "cache is advisory" policy
* documented in `docs/plans/2026-05-13-store-cache-design.md` (Write path and
* error policy):
* Write semantics (`set` / `delete`) follow the "cache is advisory" policy:
* - Backing first. If `backing.set`/`backing.delete` throws, the wrapper
* throws and the cache layer is never touched. The backing is the source
* of truth and its failures must surface to the caller.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,13 @@ describe('rudderstack retry within config.timeout', () => {
const plane = await dataPlane([503, 503, 200]);
const collector = await flowWith(plane.url);

const started = Date.now();
const result = await collector.push(createEvent({ name: 'product view' }));

// The collector races the push against config.timeout, so ok means the
// SDK's retries landed inside it; a wall clock here would also time the
// collector and the test runner.
expect(result.ok).toBe(true);
expect(plane.requests()).toBe(3);
expect(Date.now() - started).toBeLessThan(TIMEOUT);
await collector.command('shutdown');
await plane.close();
});
Expand Down
107 changes: 107 additions & 0 deletions packages/server/sources/express/src/__tests__/client-ip.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
import { startFlow } from '@walkeros/collector';
import { isObject, isString, Source } from '@walkeros/core';
import type {
Destination,
Ingest,
Mapping,
Source as SourceNS,
} from '@walkeros/core';
import { sourceExpress } from '../index';
import type { ExpressSource, Types } from '../types';

function serverOf(
source: SourceNS.Instance<Types>,
): ExpressSource['server'] | undefined {
const candidate: { type: string; server?: ExpressSource['server'] } = source;
return candidate.server;
}

// The documented one-liner for a proxy that appends `<client-ip>, <proxy-ip>`:
// the entry one hop in from the right of X-Forwarded-For.
const secondFromRight: Mapping.Fn = (scope) => {
if (!isObject(scope) || !isObject(scope.headers)) return undefined;
const chain = scope.headers['x-forwarded-for'];
return isString(chain) ? chain.split(',').at(-2)?.trim() : undefined;
};

/**
* One request to a started Express source on a real collector, with
* `config.ingest.map.ip` set to the given mapping; returns the ingest the
* destination receives.
*/
async function ingestFor(
ip: Mapping.Value,
headers: Record<string, string>,
): Promise<Ingest> {
const received: Ingest[] = [];
const recorder: Destination.Instance = {
type: 'recorder',
config: {},
push: (_event, context) => {
received.push(context.ingest);
},
};
const { collector } = await startFlow({
sources: {
express: {
code: sourceExpress,
config: {
async: false,
settings: { port: 0, paths: ['/collect'] },
ingest: { map: { ip } },
},
},
},
destinations: { recorder: { code: recorder } },
});
const server = serverOf(Source.getSource<Types>(collector, 'express'));
try {
const address = server?.address();
if (!address || typeof address === 'string') {
throw new Error('Server did not bind');
}
const response = await fetch(`http://127.0.0.1:${address.port}/collect`, {
method: 'POST',
headers: { 'content-type': 'application/json', ...headers },
body: JSON.stringify({ name: 'page view' }),
});
expect(response.status).toBe(200);
} finally {
await new Promise<void>((resolve) => {
if (!server) return resolve();
server.close(() => resolve());
});
}
expect(received).toHaveLength(1);
return received[0];
}

describe('client IP behind a proxy, as the Express docs map it', () => {
it.each([
[
'a single-address header the proxy sets',
{ key: 'headers.x-client-ip' },
{ 'x-client-ip': '203.0.113.7' },
'203.0.113.7',
],
[
'the second entry from the right of X-Forwarded-For',
{ fn: secondFromRight },
{ 'x-forwarded-for': '198.51.100.9, 203.0.113.7, 10.0.0.1' },
'203.0.113.7',
],
])('reads %s into ingest.ip', async (_, ip, headers, expected) => {
const ingest = await ingestFor(ip, headers);

expect(ingest.ip).toBe(expected);
});

it('leaves ingest.ip absent when X-Forwarded-For has a single entry', async () => {
const ingest = await ingestFor(
{ fn: secondFromRight },
{ 'x-forwarded-for': '10.0.0.1' },
);

expect(ingest).not.toHaveProperty('ip');
});
});
5 changes: 2 additions & 3 deletions packages/server/transformers/bot/src/data/crawlers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,8 @@
*
* A UA match is a claim, not proof: Screaming Frog ships Googlebot and Bingbot
* presets, and any client can send any UA. Vendor IP ranges and reverse DNS
* suffixes for the entries that publish them are listed in
* `docs/research/2026-08-20-search-crawler-list.md` and are consumed once
* identity verification exists.
* suffixes, for the entries that publish them, are in the vendor docs below
* and will be used once identity verification exists.
*
* Vendor docs of record (verified 2026-08):
* Google: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
import { startFlow } from '@walkeros/collector';
import type { Destination, Logger, Mapping, WalkerOS } from '@walkeros/core';
import { Level } from '@walkeros/core';
import { transformerFingerprint } from '../transformer';

const chrome124 =
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.6367.60 Safari/537.36';

interface Line {
level: Logger.Level;
message: string;
context: Logger.LogContext;
}

/**
* The fingerprint transformer in a destination's before chain of a real
* collector: two events pushed, every log line from WARN up recorded, and
* what the destination receives.
*/
async function deliver(
ip: Mapping.Value,
userAgent = chrome124,
data: WalkerOS.Properties = {},
) {
const lines: Line[] = [];
const received: WalkerOS.Event[] = [];
const recorder: Destination.Instance = {
type: 'recorder',
config: {},
push: (event) => {
received.push(event);
},
};
const { elb } = await startFlow({
logger: {
level: 'WARN',
handler: (level, message, context) => {
lines.push({ level, message, context });
},
},
transformers: {
fingerprint: {
code: transformerFingerprint,
config: {
settings: {
salt: 'test-salt',
ip,
userAgent: { value: userAgent },
},
},
},
},
destinations: { recorder: { code: recorder, before: 'fingerprint' } },
});
await elb({ name: 'page view', data });
await elb({ name: 'page view', data });

const errors = lines.filter((line) => line.level === Level.ERROR);
const ipEmptyWarnings = lines.filter(
(line) =>
line.level === Level.WARN &&
line.message.includes('Fingerprint input "ip" resolved empty'),
);
const hashes = received.map((event) => typeof event.user.hash);
return { errors, ipEmptyWarnings, hashes };
}

describe('an ip input that is not one address, through a real collector', () => {
it.each([
['a forwarded chain', '203.0.113.7, 172.70.1.2', ['203.0.113', '172.70']],
['a word', 'unknown', ['unknown']],
])(
'reports %s once and still delivers both events with a hash',
async (_, ip, fragments) => {
const { errors, ipEmptyWarnings, hashes } = await deliver({ value: ip });

expect(hashes).toEqual(['string', 'string']);
expect(errors).toHaveLength(1);
const [error] = errors;
expect(error.message).toContain('"ip"');
for (const fragment of fragments) {
expect(error.message).not.toContain(fragment);
expect(JSON.stringify(error.context)).not.toContain(fragment);
}
expect(error.context).toEqual({
step: 'transformer.fingerprint',
code: 'invalid',
});
expect(ipEmptyWarnings).toHaveLength(0);
},
);

it.each([
['IPv4', '203.0.113.7'],
['IPv6', '2001:db8:abcd:12::1'],
['IPv4-mapped IPv6', '::ffff:203.0.113.7'],
])('reports nothing for an %s address', async (_, ip) => {
const { errors, ipEmptyWarnings, hashes } = await deliver({ value: ip });

expect(hashes).toEqual(['string', 'string']);
expect(errors).toHaveLength(0);
expect(ipEmptyWarnings).toHaveLength(0);
});

it('keeps the one warning for an empty ip and reports no error', async () => {
const { errors, ipEmptyWarnings, hashes } = await deliver({ value: '' });

expect(hashes).toEqual(['string', 'string']);
expect(errors).toHaveLength(0);
expect(ipEmptyWarnings).toHaveLength(1);
});

it('reports nothing for an unknown user agent', async () => {
const { errors, hashes } = await deliver(
{ value: '203.0.113.7' },
'curl/8.5.0',
);

expect(hashes).toEqual(['string', 'string']);
expect(errors).toHaveLength(0);
});

it('names the configured path of the ip input, never its value', async () => {
const { errors } = await deliver('event.data.ip', chrome124, {
ip: '203.0.113.7, 172.70.1.2',
});

expect(errors).toHaveLength(1);
expect(errors[0].message).toMatch(/ \(event\.data\.ip\)$/);
expect(errors[0].message).not.toContain('203.0.113');
});
});
Loading
Loading