diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..bdfa1a60e --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,131 @@ +# Agent Development Guide + +**Skyrim Together** (codename TiltedEvolution) is a multiplayer mod for Skyrim Special Edition: co-op +questing, combat and exploration for 2-8 players. + +- Stack: C++20 / MSVC, xmake 3.x (deps pinned in the root `xmake.lua`, `Libraries/Tilted*` are + submodules). The UI is Angular (`Code/skyrim_ui`). +- Platforms: the client and launcher are Windows x64 only. The server also runs on Linux (amd64 and arm64), + so keep server and shared code portable. +- Game versions: SE **1.7.104** is the main target (`kSupportedGameVersions` in `Code/client/main.cpp`). + 1.6.1170 is unofficially supported. Engine code and address IDs are effectively identical. +- Branches: open PRs against `dev`. `master` is the release branch. +- Logs: `build\windows\x64\\logs`. + +## Architecture + +**Client-authoritative, the server mostly relays.** Clients run the simulation. The server keeps a light world +model (players, parties, cells, which actors and objects exist, who owns them) and decides who receives what. +It never simulates Skyrim. + +- The party leader effectively hosts the world. Quest progress, weather, time etc. come from the leader and + are applied on the members' clients. +- Every synced actor or object has one owner client that runs the real AI and physics and streams its state. + Other clients show a *remote* copy that replays it. Ownership moves when the owner leaves the cell or + disconnects. Players are always owned by their own client. +- Clients are trusted. Validate on the server only where it matters (parties, ownership, admin commands). + +**Engine integration (not an SKSE plugin).** `SkyrimTogether.exe` (`Code/immersive_launcher`) loads +`SkyrimSE.exe` into its own process, with the client (`Code/client`) linked in. `Code/client/main.cpp` loads +the Address Library, checks the game version, installs hooks, then loads SKSE if present so that other SKSE +plugins keep working. + +- Resolve game functions and globals through Address Library IDs, never raw addresses. +- Hooks are MinHook detours added with `TP_HOOK` in a static `TiltedPhoques::Initializer` at the bottom of the + engine file, calling the original through a saved pointer. Copy `Code/client/Games/Skyrim/Actor.cpp`. +- We don't use CommonLibSSE. Engine types are our own reversed headers in `Code/client/Games/Skyrim/**` + (`Actor`, `TESObjectREFR`, `ExtraData`, `Havok`...), with layouts pinned by `static_assert(offsetof(...))`/ + `sizeof`. Add members and asserts as needed. A local CommonLibSSE checkout is fine as a reference; in its + `Offsets*.h` the 1.6.x IDs are in the first `#ifdef SKYRIM_SUPPORT_AE` block. +- `GameFiles/Skyrim` holds what ships alongside the binaries: the ESPs, Papyrus scripts, meshes and behavior + files. + +**Data flow.** +1. Upstream: an engine hook or per-frame service poll notices a change. Hooks often run on engine worker + threads, so queue the work onto the main thread through the runner service. A client service then sends a + request through the transport service. +2. Server: a service handles the request with the sending player, updates its world model and relays a + `Notify*` to everyone, players in range or the party, usually excluding the sender. +3. Downstream: a client service subscribed to the `Notify*` applies it to the game. While applying remote + changes, guard against local hooks echoing them back upstream (existing services show how). + +**ECS (`entt`).** Client and server each have a `World` (an `entt::registry` subclass) and an `entt::dispatcher`. + +- **Services** (`Code/{client,server}/Services`): long-lived singletons in `World::ctx()` holding feature + logic. They connect in their constructors (`aDispatcher.sink().connect<&Svc::Fn>(this)`) and keep the + connections in `entt::scoped_connection` members. +- **Components** (`Code/{client,server}/Components`): plain data. Data that belongs to an actor or object + goes in a component on its entity. Use a service-side map keyed by form ID only if there's no other + option. On the client, `FormIdComponent` links an entity to its form and `LocalComponent`/`RemoteComponent` + mark the owner side. +- **Systems** (`Code/client/Systems`): near-stateless helpers over components (interpolation, animation...). +- **Events** (`Code/{client,server}/Events`): small structs passed through the dispatcher. + +## Network protocol + +- Transport is `Libraries/TiltedConnect` (Valve GameNetworkingSockets). All messages live in `Code/encoding` + (`SkyrimEncoding`), shared by client and server. New messages follow the existing ones' naming, structure + and registration. +- Reuse payload types from `Code/encoding/Structs`: `GameId` (`ModId` + `BaseId`, **never raw runtime form + IDs**), `Vector3_NetQuantize`, `Rotator2_NetQuantize`. +- **No backward compatibility:** the client sends `BUILD_COMMIT` in `AuthenticationRequest` and the server + rejects any mismatch (`kWrongVersion`), so reorder or change opcodes and fields freely. +- Watch bandwidth. Sends are reliable by default; send high-rate, loss-tolerant data (movement) unreliably. + +## Building + +From the repo root (VS 2022 developer shell or PowerShell with MSVC): + +```sh +xmake f -m releasedbg -y # configure (debug | releasedbg | release); first run fetches deps +xmake -y # build everything +xmake build SkyrimTogetherClient # one target; also SkyrimImmersiveLauncher, SkyrimTogetherServer +xmake build TPTests && xmake run TPTests # main unit tests (Catch2) +xmake project -k vsxmake # regenerate the VS solution; rerun after adding/removing files +``` + +- The solution is `vsxmake2022/.sln`. Most contributors build and debug from it; agents don't need it. +- UI: `cd Code/skyrim_ui && pnpm install && pnpm deploy:production`. `xmake install -o distrib` gathers + binaries. +- Main targets: + - `SkyrimImmersiveLauncher` → `SkyrimTogether.exe`, with the `SkyrimTogetherClient` static lib linked in + - `SkyrimTogetherServer` → `STServer.dll`, run by `SkyrimServerRunner` → `SkyrimTogetherServer.exe` + - `ImmersiveElf` → `EarlyLoad.dll` (launcher early loader), `TPProcess` (UI overlay helper process) + - Shared static libs: `SkyrimEncoding`, `CommonLib`, `BaseLib` + - Tests: `TPTests`, plus `Console_Tests`, `ESLoader_Tests` and `Resources_Tests` for the component libs + in `Code/components` + +## Testing + +- **Most gameplay features can only be verified by hand:** run a local server plus two or more game clients + on one machine and exercise the feature from both sides. Say plainly when a change still needs an + in-game test. Compiling is not proof that it works. +- Add unit tests only for mission-critical, Skyrim-independent code: protocol encoding, timing and + interpolation math, pure data structures. Such code can be compiled into `TPTests`. Ask before writing + tests for gameplay code. + +## Reverse engineering + +- **IMPORTANT: if a task depends on engine behavior the source doesn't show (hook points, struct layouts, + vtable slots, threading) and no IDA or Ghidra MCP tools are available, stop before implementing.** + Say what needs verifying in the binary, warn that a source-only version may be wrong or suboptimal (guessed + offsets, wrong hook point, missed side effects), and ask whether to go ahead. +- With an MCP, check the Skyrim SE 1.6.1170 database first. Use 1.7.104 only when something differs. +- Map Address Library IDs to IDB addresses and back with `python Tools/Scripts/addrlib.py ...` + (1.6.1170 by default, `-v 1.7.104` for another version; usage at the top of the script). +- `Tools/Scripts` has other helpers for addresses, RTTI dumps and IDB matching. + +## Conventions + +- **Match the surrounding style.** Hungarian-style prefixes where the file uses them: `a` arguments, `c` const, + `p` pointers, combined as needed (`acEvent`, `apActor`). No `r` prefix for references. +- Members use `m_` (plain-data structs use bare `PascalCase` fields). Statics use `s_`, globals `g_`; hook originals are + `Real`, constants `k`. Types and functions are `PascalCase`, and most functions are `noexcept`. + Reversed engine types keep the game's names (`hCaster`, `slotType`). +- No exceptions and no throwing STL calls. No `auto` for primitive types. +- Use `TiltedPhoques` containers (`String`, `Vector`, `UniquePtr`...) where the surrounding code does. Log + with `spdlog`. +- Gate debug-only UI and behavior with `IS_MASTER` / `DEBUG`. Mod compatibility shims live in + `Code/client/ModCompat`. +- Commits: short imperative title describing the user-visible effect ("Fix X when Y", "Stop X doing Y"), and + a short body explaining why. diff --git a/CODE_GUIDELINES.md b/CODE_GUIDELINES.md index 57eacd289..127378b32 100644 --- a/CODE_GUIDELINES.md +++ b/CODE_GUIDELINES.md @@ -2,7 +2,7 @@ ## Language -We are using C++20, any C++20 feature supported by vs2019 is allowed. +We are using C++20, any C++20 feature supported by vs2022 is allowed. Please try to use templates responsibly, we don't want compilation times to explode and to deal with bloated binaries. @@ -45,10 +45,3 @@ Names must be self explanatory, ``size_t a;`` is not acceptable, ``size_t incomi Don't use java style blocks, a ``{`` needs to be on a new line. Don't use exceptions, don't use STL code that can throw, use the nothrow version if available or the unsafe version. - -## Commit Naming -Preface commit messages with one of the following tags: -* `feat:` for new features -* `tweak:` for tweaks to an existing system -* `fix:` for bug fixes and crash fixes -* `refactor:` for refactoring \ No newline at end of file diff --git a/README.md b/README.md index 707806163..c4ebb1efa 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,16 @@ -# Tilted Online +# Skyrim Together +#### codename "TiltedEvolution" ![Build status](https://github.com/tiltedphoques/TiltedEvolution/workflows/Build%20windows/badge.svg?branch=master) [![Build linux](https://github.com/tiltedphoques/TiltedEvolution/actions/workflows/linux.yml/badge.svg)](https://github.com/tiltedphoques/TiltedEvolution/actions/workflows/linux.yml) [![Discord](https://img.shields.io/discord/247835175860305931.svg?label=&logo=discord&logoColor=ffffff&color=7389D8&labelColor=6A7EC2)](https://discord.gg/skyrimtogether) Size Limit logo by Anton Lovchikov -Tilted Online is a framework created to enable multiplayer in Bethesda games, currently supporting **Skyrim Special Edition**. +TiltedEvolution is a framework created to enable multiplayer in Bethesda games, currently supporting **Skyrim Special Edition**. ## Getting started -To play Tilted Online, go to the [nexus page](https://www.nexusmods.com/skyrimspecialedition/mods/69993). +To play Skyrim Together, go to the [nexus page](https://www.nexusmods.com/skyrimspecialedition/mods/69993). -For general information, go to the [Tilted Online Wiki](https://wiki.tiltedphoques.com/tilted-online/). +For general information, go to the [TiltedEvolution Wiki](https://wiki.tiltedphoques.com/tilted-online/). Check out the [build guide](https://wiki.tiltedphoques.com/tilted-online/technical-documentation/build-guide) for setup and development info on the project. When writing code, check the CODE_GUIDELINES.md and make sure to run clang-format! @@ -20,24 +21,30 @@ If you would like to report a bug please report them in the "Issues" tab on this Have some experience in C++, and want to help advance the project faster? Contribute! - Check the issues, it's a good place to start when you don't know what to do. - Fork the repository and create pull requests to this repository. -- Create pull requests to the "dev" branch, not to master or prerel. +- Create pull requests to the "dev" branch, not to master. - Try to keep your code clean, following the code guidelines. ## Main project source tree -* [**client/**](./Code/client): Sources for the SkyrimSE and FO4 clients. -* [**immersive_launcher/**](./Code/immersive_launcher): Game starter/updater. -* [**common/**](./Code/common): Common code shared between plugin and server. -* [**encoding/**](./Code/encoding): Net-message definitions. -* [**server/**](./Code/server): GameServer implementation. -* [**skyrim_ui/**](./Code/skyrim_ui): Source code for the ui, written in typescript. -* [**tests/**](./Code/tests): Tests for the encoding and serialization code. -* [**tp_process/**](./Code/tp_process): Worker for CEF (Chromium Embedded Framework) overlay. +* [**client/**](./Code/client): Skyrim SE client: engine hooks, reverse-engineered game types and the sync logic. +* [**immersive_launcher/**](./Code/immersive_launcher): `SkyrimTogether.exe`, which starts the game with the client loaded. +* [**immersive_elf/**](./Code/immersive_elf): `EarlyLoad.dll` helper DLL for the launcher. Detects if Mod Organizer 2 was injected first. +* [**tp_process/**](./Code/tp_process): Worker for the CEF (Chromium Embedded Framework) overlay. +* [**skyrim_ui/**](./Code/skyrim_ui): In-game UI, written in Angular/TypeScript. +* [**server/**](./Code/server): Game server (`STServer.dll`). +* [**server_runner/**](./Code/server_runner): Dedicated server executable that hosts the server DLL. +* [**encoding/**](./Code/encoding): Network message and struct definitions shared by the client and server. +* [**common/**](./Code/common), [**base/**](./Code/base): Shared utilities. +* [**components/**](./Code/components): Shared modules (console/settings, ESP loader, resources, ImGui). +* [**tests/**](./Code/tests): Unit tests. +* [**GameFiles/**](./GameFiles): Plugins, scripts and assets shipped with the mod. +* [**Libraries/**](./Libraries): Networking, hooking, reverse-engineering and UI libraries (git submodules). +* [**Tools/**](./Tools): Reverse-engineering and address scripts. ## License [![GNU GPLv3 Image](https://www.gnu.org/graphics/gplv3-127x51.png)](http://www.gnu.org/licenses/gpl-3.0.en.html) -Tilted Online is free software: you can redistribute it and/or modify +TiltedEvolution is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. diff --git a/Tools/Scripts/address_patcher.py b/Tools/Scripts/address_patcher.py deleted file mode 100644 index 4ad63be4d..000000000 --- a/Tools/Scripts/address_patcher.py +++ /dev/null @@ -1,43 +0,0 @@ -import os -from pathlib import Path - -# run this script from Tools/Scripts/, and put the 'new_addresses.txt' file there -rootdir = Path(__file__).parent.parent.parent.joinpath("Code").joinpath("client") - -# collect all paths in the Client/ folder -paths = [] - -for subdir, dirs, files in os.walk(rootdir): - for file in files: - if ".h" in file or ".cpp" in file: - paths.append(os.path.join(subdir, file)) - -with open('new_addresses.txt') as new_addresses: - for address_pair in new_addresses: - address_pair = address_pair.strip().split(',') - - is_address_rewritten = False - - for path in paths: - with open(path, 'r') as source_file: - file_data = source_file.read() - - if address_pair[0] in file_data: - print("found address " + address_pair[0]) - - file_data = file_data.replace(address_pair[0], address_pair[1]) - - with open(path, 'w') as source_file: - source_file.write(file_data) - - is_address_rewritten = True - # don't break cause the address might be in multiple places - # it obviously shouldnt be, but it be like that rn - #break - - if not is_address_rewritten: - print("Address was not found and has not been rewritten! " + address_pair[0]) - continue - - -print("Done patching!") diff --git a/Tools/Scripts/addrlib.py b/Tools/Scripts/addrlib.py new file mode 100644 index 000000000..dafda92d7 --- /dev/null +++ b/Tools/Scripts/addrlib.py @@ -0,0 +1,107 @@ +"""Map Skyrim Address Library IDs <-> addresses (format follows Code/client/VersionDb.h). + + python Tools/Scripts/addrlib.py 39004 0x1406cf760 ID -> address, address -> ID + python Tools/Scripts/addrlib.py -v 1.7.104 39004 other game version (default 1.6.1170, the main IDB) + +Addresses are 0x140000000 + offset, as in the IDBs; the 0x is optional for 9+ digit addresses. +An address inside a function or object resolves to the closest ID below it, e.g. "39004+0x1a". +-v also takes a path to a .bin. Bins are read from /Data/SKSE/Plugins; the game dir comes +from SKYRIM_SE_DIR or the Steam registry key. +""" + +import bisect +import os +import struct +import sys + +BASE = 0x140000000 + +# Format 2 delta codes: kind -> (stored value, sign). Sign 0 means the value replaces the +# previous one, otherwise it's added/subtracted. Kind 1 is "previous + 1" with nothing stored. +DELTAS = {0: ("") + parts = (version.split(".") + ["0"] * 4)[:4] + path = os.path.join(game, "Data", "SKSE", "Plugins", f"versionlib-{'-'.join(parts)}.bin") + if not os.path.isfile(path): + sys.exit(f"{path} not found (is Address Library for {version} installed?)") + return path + + +def delta(d, p, kind, prev): + """Decodes one format 2 value at d[p]. Returns (value, next p).""" + if kind == 1: + return prev + 1, p + fmt, sign = DELTAS[kind] + v = struct.unpack_from(fmt, d, p)[0] + return prev + sign * v if sign else v, p + struct.calcsize(fmt) + + +def load(path): + """Returns {id: offset}.""" + with open(path, "rb") as f: + d = f.read() + fmt = struct.unpack_from("> 4) & 7, off // scale) + off *= scale + out[vid] = off + return out + + +def main(): + args = sys.argv[1:] + version = "1.6.1170" + if args[:1] == ["-v"]: + version, args = args[1], args[2:] + if not args or args[0] in ("-h", "--help"): + sys.exit(__doc__) + ids = load(find_bin(version)) + rev = sorted((o, i) for i, o in ids.items()) + + for arg in args: + if not (arg.lower().startswith("0x") or len(arg) >= 9): + o = ids.get(int(arg)) + print(f"{arg} -> {o + BASE:#x}" if o is not None else f"{arg} -> not found") + continue + target = int(arg, 16) - BASE + n = bisect.bisect_right(rev, (target, float("inf"))) - 1 + if n < 0 or target - rev[n][0] > 0x1000000: # outside the image (largest real gap is 0x1000000) + print(f"{arg} -> not found") + else: + o, i = rev[n] + print(f"{arg} -> {i}" + (f"+{target - o:#x}" if target != o else "")) + + +if __name__ == "__main__": + main() diff --git a/Tools/Scripts/addrlib_conv.py b/Tools/Scripts/addrlib_conv.py deleted file mode 100644 index 1e9994d0c..000000000 --- a/Tools/Scripts/addrlib_conv.py +++ /dev/null @@ -1,51 +0,0 @@ -addresses = {} - -with open('out.txt') as f: - lines = f.readlines() - for l in lines: - split_string = l.split("\t") - addresses[split_string[1][:-1]] = split_string[0] - -import glob -import re - -def handle_file(filename): - if filename == 'Utils.h' or filename == "EventDispatcher.h": - return - - with open(filename, 'r') as file : - filedata = file.read() - - replace = {} - pointers = [m.start() for m in re.finditer('POINTER_FALLOUT4', filedata)] - if len(pointers) > 0: - for p in pointers: - start_loc = filedata.find('0x', p) - mid_loc = filedata.find(' ', start_loc) - end_loc = filedata.find(')', start_loc) - - addr = filedata[start_loc:mid_loc].lower() - try: - id = addresses[addr] - replace[filedata[start_loc:end_loc]] = id - except: - print("Missing " + addr) - - #filedata.find( 'POINTER_SKYRIMSE') - - if len(replace) > 0: - for fr in replace: - filedata = filedata.replace(fr, replace[fr]) - - with open(filename, 'w') as file: - file.write(filedata) - -# root_dir needs a trailing slash (i.e. /root/dir/) -for filename in glob.iglob('**/*.cpp', recursive=True): - handle_file(filename) - -for filename in glob.iglob('**/*.hpp', recursive=True): - handle_file(filename) - -for filename in glob.iglob('**/*.h', recursive=True): - handle_file(filename) diff --git a/Tools/Scripts/get_skyrim_addresses.py b/Tools/Scripts/get_skyrim_addresses.py deleted file mode 100644 index 90012d394..000000000 --- a/Tools/Scripts/get_skyrim_addresses.py +++ /dev/null @@ -1,44 +0,0 @@ -import os -from pathlib import Path - -# run this script from Tools/Scripts/ -rootdir = Path(__file__).parent.parent.parent.joinpath("Code").joinpath("client") - -# collect all paths in the Client/ folder -paths = [] - -for subdir, dirs, files in os.walk(rootdir): - for file in files: - if ".h" in file or ".cpp" in file: - paths.append(os.path.join(subdir, file)) - -with open('skyrim_addresses.txt', 'w') as output_file: - for path in paths: - with open(path, 'r') as source_file: - for line in source_file: - if "POINTER_SKYRIMSE" in line: - if "#define POINTER_SKYRIMSE" in line: - print("Define found, skipping line.") - continue - - begin_pos = line.find("0x14") - if begin_pos == -1: - print(f"Something went wrong in file {file}, line {line}") - continue - - end_pos = begin_pos + 11 - address = line[begin_pos:end_pos] - output_file.write(address + "\n") - -lines_seen = set() - -with open('skyrim_addresses.txt', 'r+') as output_file: - lines = output_file.readlines() - output_file.seek(0) - for line in lines: - if line in lines_seen: - print(f"duplicate! {line}") - else: - output_file.write(line) - lines_seen.add(line) - output_file.truncate() \ No newline at end of file