AI Gauntlet for Unreal Engine

The short version: enable the plugin, open Claude Code, Cursor, Antigravity or Codex in your project folder and paste the one-paragraph prompt in the first section. The assistant starts the game itself with aigauntlet.py launch, reads Plugins/AIGauntlet/AGENTS.md, drives the game and stops it. You do not start anything and you do not need to learn the verbs. Everything after that section is the reference for doing it by hand.

The same pages are available inside the editor under Tools > AI Gauntlet > AI Gauntlet Documentation.

Use it with a coding assistant

This is the intended way to use AI Gauntlet, and the short one. The plugin was built for a coding assistant to drive - Claude Code, Cursor, Antigravity, Codex, or any agent that can run a shell command - and it ships the instructions the assistant needs inside the plugin folder. You do not have to learn the verbs. The assistant does.

If you would rather drive it by hand, every page in Docs/ still applies; skip to Install.md.

Two steps

1. Enable the plugin and build, as for any code plugin. Edit → Plugins → AI Gauntlet, tick it, restart the editor, let it compile.

2. Open your coding assistant in the project folder and paste this:

Read Plugins/AIGauntlet/AGENTS.md. Launch my game with python Plugins/AIGauntlet/Tools/aigauntlet.py launch, run orient, tell me what you see, then: play from the main menu into the first level and report anything that looks wrong. Stop the game when you are done.

Replace the italic part with whatever you want done. That is the whole setup. You do not start the game: launch finds your .uproject, resolves the engine it is associated with, starts the editor in game mode with -aigauntlet, and returns when the socket answers. For a packaged build the assistant passes --exe path\to\MyGame.exe instead, and stop ends whatever launch started, by pid.

What the assistant reads

AGENTS.md and CLAUDE.md at the plugin root are the same document under the two names assistants look for. Claude Code picks up CLAUDE.md from any folder it works in; Cursor, Antigravity, Codex and most others read AGENTS.md. Either way the agent learns:

• the one command to run first (orient), and how to read what it prints;
• every verb, and the three that keep a session fast (batch, input.click, wait.until);
• how to find things in a game it has never seen - tags, not names; widgets.find for UI;
• the traps, in the order it will hit them: arrays that replace rather than merge, the input route that reaches menus and the two that do not, a deferred GPU readback that reports the previous frame, a solve that advances the game before you can inspect it.

None of that needs a prompt from you. The document is the prompt.

What to ask for

Things people have actually asked an assistant to do through this plugin, all of which it did with no game-side code written:

• "Play the campaign from the start and stop at the first level you cannot pass. Screenshot it and tell me the score the game gave you." A whole sixty-level campaign was played this way.
• "Open the options screen, change every setting once, and tell me which ones did not take effect."
• "Connect a PlayStation gamepad and get from the title screen into a level using only the pad. Say where keyboard would have been needed."
• "Reproduce this bug report: [paste]. Get to the same state in the Shipping build and capture screenshot, state and log."
• "Type two player names into the party setup screen and play one round through the UI."
• "Run the same level twenty times and tell me if any number drifts."

Every one of those is a paragraph of English. The assistant turns it into verbs.

When it gets stuck

The assistant will tell you what it last saw - wait.until times out with the value it observed, object.get reports unknown property names instead of silence, log.tail says how many lines it missed. Most stalls are one of these, and the agent knows them from AGENTS.md:

• It clicks and nothing happens. The game window is not focused, or the click went in through an in-process route. window.focus, then input.click.
• A key does nothing on a screen that says "press any key". The widget reports focusable: false. That is your game's bug (SetIsFocusable(true) before SetFocus()), and the agent will point at it.
• It cannot find an actor by name. Names are internal at runtime. It should read the tag histogram from orient and use actors.find --tags.
• launch fails. It says why: no .uproject above the working directory (pass --uproject), the engine version in the project is not installed on this machine (pass --exe to an UnrealEditor.exe or a packaged game), or the game exited before the socket opened, which almost always means the plugin is not enabled in the .uproject.
• It cannot connect to a game you started yourself. It was started without -aigauntlet, or the token file is in a %LOCALAPPDATA% your Python cannot see. Install.md covers both.

Doing it by hand instead

Everything the assistant does, you can do at a terminal:

cd Plugins/AIGauntlet/Tools
python aigauntlet.py launch
python aigauntlet.py orient
python aigauntlet.py widgets.find --text START
python aigauntlet.py input.click --x 180 --y 503
python aigauntlet.py stop

Verbs.md is the reference, Protocol.md is the wire format if you are writing your own client, and UseCases.md is where each configuration earns its place.

AI Gauntlet

A localhost JSON control surface for a running Unreal Engine game, so an external AI agent or test script can inspect state, call functions, read the log, drive the UI with a mouse, a keyboard or a simulated gamepad, and take screenshots — in a packaged build, including Shipping, where the console does not exist.

Why it exists

Unreal already has good automation for the editor. What it does not have is a way for something outside the process to play a shipped build. -ExecCmds needs a console, and Shipping strips it. Gauntlet can launch and watch a build, but the thing that presses the buttons still has to live inside the game.

AI Gauntlet is that inside piece, exposed over a socket. It never touches the console, so it works in every build configuration.

It is not only for packaged builds. The same plugin and the same verbs run in an editor game-mode build (UnrealEditor.exe MyProject.uproject -game -aigauntlet), which is where the fast work happens: uncooked content, no cook, no package, and a Development build that actually logs. Package when you want to verify; use the editor while you are still writing. See UseCases.md.

🔴 One rule before anything else

Do not ship this plugin enabled in a build you send to a store. Untick AI Gauntlet in Edit → Plugins before you build a release; a disabled plugin is not compiled and not staged, so nothing of it reaches your players.

There is one control and that is it — the plugin's own checkbox. The surface compiles into every configuration, Shipping included, because a Fab code plugin arrives precompiled and a compile-time switch would have locked Blueprint-only projects out of Shipping entirely. What protects a build you forgot to untick is the launch flag and the token: no -aigauntlet, no socket. See Security.md.

Quickstart, with a coding assistant (the intended way)

You do not need to learn the verbs. The assistant does - the instructions it needs ship in this folder as AGENTS.md (Cursor, Antigravity, Codex, and most others) and CLAUDE.md (Claude Code), which are the same document.

1. Enable the plugin and build.
2. Open your assistant in the project folder and paste:

Read Plugins/AIGauntlet/AGENTS.md. Launch my game with python Plugins/AIGauntlet/Tools/aigauntlet.py launch, run orient, tell me what you see, then: play from the main menu into the first level and report anything that looks wrong. Stop the game when you are done.

The assistant starts the game itself: launch finds the .uproject, resolves its engine, and runs the editor in game mode with -aigauntlet (or a packaged exe with --exe).

Swap the italic part for the task you want. WithAnAssistant.md has more to ask for and what to do when it stalls.

Quickstart, by hand

## 1. Copy this folder into your project's Plugins/, enable it in the .uproject, build.
## 2. Start it - the editor in game mode, or --exe for a packaged build:
cd Plugins/AIGauntlet/Tools
python aigauntlet.py launch

## 3. Look around:
python aigauntlet.py orient

orient prints the map, the framework classes with their callable APIs, what is on screen, the actor tag histogram and any recent warnings. It is the right first command against a game nobody has automated before.

from aigauntlet import Gauntlet

with Gauntlet() as g:
    print(g.orient())
    g.click_widget("START")        # by the label, not by a coordinate
    print(g.log_tail(limit=20)["lines"])
    print(g.screenshot("after_start"))

No game-side code is required. The reflection layer covers any project; one commercial game's whole campaign was driven through object.call and object.get alone.

Documentation

Also inside the editor: Tools > AI Gauntlet > AI Gauntlet Documentation opens every page below in a tab, with buttons to open the Tools folder and copy the quickstart command.

Limits worth knowing before you buy

• It cannot attach to an already-built exe. The plugin has to be compiled in, which means one repackage. After that it is drop-in for every future build.
• Reflection only. Plain C++ members that are not UPROPERTY do not exist to this plugin. Neither do functions that are not UFUNCTION.
• The os input route is Windows-only in this version, and it is the only route that reaches Slate UI. The pc and slate routes work everywhere. Verbs fail loudly off Windows rather than pretending.
• -nullrhi has no viewport, so screenshot fails there and anything needing the renderer will not work. Run windowed and offscreen instead of truly headless.
• Loopback plus a token is the whole security model. Do not port-forward it, and do not ship a build that passes -aigauntlet by default.
• One world. Verbs act on the active game instance's world. Multi-client tests need one process, and one port, each.

Harsh Bakshi · skrots.com

Install and run

Add the plugin

1. Copy this folder into your project's Plugins/ directory.
2. Add it to your .uproject:

{ "Name": "AIGauntlet", "Enabled": true }

1. Build. No game-side code is required — the reflection layer covers any project.

Windows only for this version. See InputRoutes.md for exactly which parts are platform-specific and which are not.

Then hand it to your coding assistant

That is the intended use, and the rest of this page is only needed if you want to drive it by hand. Open Claude Code, Cursor, Antigravity or any shell-capable assistant in the project folder and paste the prompt in WithAnAssistant.md. The assistant reads Plugins/AIGauntlet/AGENTS.md, starts the game itself with aigauntlet.py launch, and knows the surface. You do not need to start anything.

Run

The quick way: no packaging at all

You do not have to package anything to use this. The editor binary in game mode opens the same socket, against uncooked content:

UnrealEditor.exe MyProject.uproject -game -aigauntlet -windowed -ResX=1280 -ResY=720

That is a Development build, so a content change is a restart rather than a cook, and the log is intact. It is the right place for the iteration work; package when you want to verify against cooked content and real load order. See UseCases.md.

The packaged way

The server is off unless asked for. A shipped game with the plugin compiled in and no flag on the command line opens no socket at all.

MyGame.exe -aigauntlet
MyGame.exe -aigauntlet -aigport=7799

It binds 127.0.0.1 only. The address is deliberately not configurable: this is a remote-function-call surface, and it has no business on a network.

On startup the game writes a connection token to <ProjectSaved>/AIGauntlet/<pid>.token and logs the path. The Python client finds it automatically — you should never need to look.

Watch for these two lines:

LogAIGauntlet: Display: Token written to .../Saved/AIGauntlet/12345.token
LogAIGauntlet: Display: Listening on 127.0.0.1:7788

Documentation inside the editor

With the plugin enabled, Tools > AI Gauntlet > AI Gauntlet Documentation opens this page and every other one in an editor tab (also under Window > Developer Tools). It reads the Markdown files shipped in the plugin folder, so what you see there is what you are reading now.

Talk to it

The client is a single Python file with no dependencies beyond the standard library. It needs Python 3.10 or newer, which the plugin does not bundle: install it from https://www.python.org/downloads/ (tick "Add python.exe to PATH"). Any other language that can open a TCP socket and write a line of JSON works too - see Protocol.md; Python is only the one that ships.

cd Plugins/AIGauntlet/Tools
python aigauntlet.py launch     # starts the game (editor game mode, or --exe MyGame.exe), waits for the socket
python aigauntlet.py orient
python aigauntlet.py stop       # ends what launch started

launch finds the .uproject above it, resolves the engine from EngineAssociation (Launcher install, registry, or a source-build GUID) and appends -aigauntlet -aigport -windowed -ResX -ResY; --map, --res, --port and any extra flags pass through. --dry-run prints the command it would run. orient is the right first command against a game you do not know: it prints the map, the framework classes with how many functions each exposes, what is on screen, which actor tags exist, and whether the log is complaining.

Shipping builds, and the one rule

The plugin compiles into every configuration, Shipping included. There is no second switch and nothing to opt into: if the plugin is ticked, the surface is in the build.

That is deliberate. A Fab code plugin arrives precompiled — Epic's build farm compiles it, not you — so any compile-time decision is already baked by the time it reaches your project, and a Blueprint-only project has no way to rebuild it. A compile-time gate would have meant "no Shipping support unless you can compile C++", which is most of the audience.

So driving a Shipping build needs nothing special:

RunUAT.bat BuildCookRun -project=... -clientconfig=Shipping -build -cook -stage -pak
MyGame-Shipping.exe -aigauntlet

🔴 Do not ship this plugin enabled in a build you send to a store

Untick AI Gauntlet in Edit → Plugins before you build a release. That one checkbox is the whole control: a disabled plugin is not compiled and not staged, so nothing of it reaches your players.

What protects a build you forgot to untick is the launch flag and the token — no -aigauntlet on the command line, no socket, and no verb runs before a client proves it can read a user-scoped token file. That is real protection, and it is not a substitute for the checkbox. Put "AI Gauntlet unticked" on your release checklist next to "shipping config".

Two things make the checkbox easy to verify:

• Plugin enablement is recorded in your .uproject, so it shows up in git status and in a diff like any other change.
• A build with the plugin disabled contains no AIGauntlet module at all — check your packaged Binaries folder if you want to see it for yourself.

If you want a hard compile-out anyway

The source is guarded by AIGAUNTLET_ENABLED, defined to 1 by the plugin's Build.cs. Set it to 0 in your own target rules and the listener, the verbs, the reflection bridge and the OS input layer all compile to nothing. That is an option for teams building from source, not the default, and it does nothing for a precompiled install — which is exactly why it is not the primary mechanism.

See Security.md for what the flag and the token do and do not protect against.

Where to use it

The headline is "drive a packaged Shipping build", because that is the thing nothing else does. It is not the only thing, and for most teams it is not even the daily one.

Every configuration is the same plugin and the same verbs. What changes is where you run it.

On PIE: the listener lives on a UGameInstanceSubsystem, so it starts whenever a game instance does — which includes pressing Play in the editor, provided -aigauntlet was on the editor's command line. That follows from how it is built rather than from a test anyone has run, so it is marked as such. Editor game mode is the one that was actually driven.

One consequence worth knowing either way: where the token file lands depends on how you launched. Run from the project and it is <Project>/Saved/AIGauntlet; a staged build puts it under %LOCALAPPDATA%/<ProjectName>/Saved. The client looks in both.

The inner loop: test while you are still writing it

You do not need to package anything to use this. Run the editor binary in game mode and the socket is there:

UnrealEditor.exe MyProject.uproject -game -aigauntlet -windowed -ResX=1280 -ResY=720
python aigauntlet.py orient

That is a Development build against uncooked content, so a content change is a restart rather than a cook. Most of the scripting work on this plugin's own test suite was done that way, and only the final verification ran against a package.

Two things make this the right place for the fast work:

• snapshot and widgets.find tell you what your own UI is doing, which is often faster than reading it back out of your own code. focusable: false on a screen you just wrote is a one-call answer to "why is my keyboard handler not firing".
• log.tail is at its most useful here, because a Development build actually logs. Measured on the same game, same script, minutes apart: 58 errors and 60 warnings from an editor game-mode build, against 5 lines and nothing at Warning or worse from the packaged Shipping build. The verb, the cursors and the filters are identical — Shipping simply has almost nothing to read, because the engine compiles most UE_LOG calls away there. Do your log-reading in Development and do not build a Shipping test that depends on log content.

Internal testing: your own regressions, on your own machine

The obvious use, and the one that pays for itself first. A script that drives your game through a real sequence catches the class of defect that compiles, cooks and packages perfectly and is still wrong.

Written against this plugin's own development, all found by driving rather than by reading:

• A results card fired a chapter early.
• A speed bonus that had been dead for some time, because the value it read was never set.
• Two identical pass marks nine decimals apart, where a mode was grading half a puzzle against the whole puzzle's threshold.
• A solve that never registered because the function meant to record it had no callers — visible immediately as a state row that disagreed with the screen.

None of those are crashes. None would fail a build. All were found by reading the game's own state through object.get and noticing it disagreed with what was on screen — which is exactly what this surface is for.

External bugs: reproducing what somebody else reported

A bug report from a player or a QA pass is usually a description, not a repro. The expensive part is getting back to the state where it happened.

A script gets you there the same way every time:

• Drive to the exact situation — level, mode, options, difficulty — in seconds rather than by playing. level.open, object.set on the settings, object.call to start the thing.
• Run it in the configuration they were on. This is where Shipping earns its place: if the report came from a shipped build, reproduce it there, not in the editor, because stripped logging, cooked assets and optimisation are all differences that hide bugs.
• Capture what you find. screenshot for the visual, log.tail for what the engine was saying, snapshot for the state around it. That is a bug report somebody can act on.
• Keep the repro. The script that reproduced it once becomes the regression test that proves it stays fixed.

The same applies to bugs that are not in your code at all — an engine warning, a plugin misbehaving, an asset that fails to load only in a cooked build. log.tail surfaces those without anyone needing to know where the log file lives, which on a staged build is not where most people would look.

Soak and stress, without a person watching

Verbs run on the game thread and nothing needs a human, so a loop can run for as long as you like:

• Play the same level a hundred times and watch for a score, a state or a memory figure that drifts.
• Travel between levels repeatedly, which is where leaks and load-order problems surface.
• Hammer a menu, then check log.tail for what the engine thought of it.

Use batch for this. A round trip costs about 17 ms, so a loop that makes six calls per iteration is spending most of its time waiting rather than testing.

CI, honestly

It works, with one caveat worth knowing before you build a pipeline on it: -nullrhi is not an option. Anything that reads the renderer — screenshots, and in this game's case its own scoring — needs a real viewport. "Headless" here means windowed and offscreen on a machine with a GPU, not no-RHI.

Given that, a runner that launches the build, runs a script and reports an exit code is straightforward, and the script is the same one you use by hand.

Where it is the wrong tool

• Unit-testing pure logic. Use the automation framework; it does not need a running game.
• Editor tooling. This drives a game, not the editor. An Editor Utility Widget is the right answer there.
• Anything not reflected. Only UFUNCTIONs and UPROPERTYs exist to it. A plain C++ member is invisible, and the way through is usually the player's own path — project a world position to the screen and click it.
• Performance profiling. Read Unreal Insights. This can tell you a level dropped frames; it cannot tell you why.

Verbs reference

34 built in. describe lists whatever is actually registered, including verbs your own game adds — trust it over this page.

Being fast

Read this before writing a loop. A round trip costs about 17 ms on loopback — measured, and not negotiable. What is negotiable is how many you make.

python aigauntlet.py batch --args '{"calls":[
  {"verb":"ping","args":{}},
  {"verb":"object.get","args":{"object":"gamemode","properties":["__class"]}},
  {"verb":"wait","args":{"frames":2}}]}'

{"results": [{"ok": true, "result": {...}}, {"ok": true, "result": {...}}, {"ok": true, "result": {}}],
 "ran": 3, "failed": 0}

Measured on a packaged Shipping build: six reads took 99 ms separately and 18 ms in one batch. Children run in order on the game thread, and a child that does not answer this tick — wait, screenshot, a typed string — is waited for before the next one starts, so a whole input gesture works inside a batch. stopOnError defaults to true; set it false to run the rest anyway. A batch cannot contain a batch.

input.click replaces the six calls a click used to take:

python aigauntlet.py input.click --x 244 --y 500

It moves the cursor, lets it settle, presses, holds, and releases — all on the game thread. The details matter and were each needed to make a Slate button respond:

• The hold is real. input.key event=tap defaults to two frames, which suits an Enhanced Input Hold/Tap trigger and is too short for a button that has just taken mouse capture.
• The cursor settles before the press, or the press can beat the hover.
• activate: true sends a throwaway click at a neutral point first, because the first click on a background window is spent activating it. Never at the target: if that press both activates and presses, the second click lands on whatever the next screen put there.

wait.until waits on the game thread instead of paying a round trip per poll:

python aigauntlet.py wait.until --object "component:gamemode/Round" --function IsRoundActive --equals true

{"satisfied": true, "observed": "true", "frames": 34}

Reads a UFUNCTION's return value or a reflected property and compares as text, so one path covers names, enums, numbers and bools. A timeout reports what it last saw, which is usually the whole diagnosis:

wait.until timed out after 30 frames; last saw 'false'

snapshot

The one to call first on a game you have never seen.

python aigauntlet.py snapshot
python aigauntlet.py orient     # the same data, rendered as a paragraph

Returns the map and viewport; the five framework objects (gameMode, gameState, playerController, playerState, pawn) each with its class name and its BlueprintCallable signatures; the visible widget tree with text and rects; a histogram of every actor tag in the level with counts; and recent warnings and errors.

depth: "full" adds property values. brief is the default because the first call should be cheap — a brief snapshot is a few kilobytes and is capped at 40 widgets, 30 properties and 20 log lines. When it clips, it says so in truncated.

The tag histogram is the field to read first. GetName() returns an internal name at runtime, so many games identify nothing by name and tags are the only handle that exists. One measured case: {"Prop": 48, "ShelfProp": 2, "Staged": 2} — which both located the objects the game cared about and revealed that 46 pieces of set dressing were carrying a tag the game's own scoring selected on.

widgets.find

python aigauntlet.py widgets.find --text START

{"object": "widget:BacklitMainMenu_C_0", "class": "BacklitMainMenu_C",
 "visible": true, "focusable": false, "hasFocus": false,
 "rect": {"x": 0, "y": 0, "w": 1280, "h": 720, "cx": 640, "cy": 360},
 "children": [
   {"class": "UButton", "name": "StartButton", "text": "START",
    "rect": {"x": 110, "y": 486, "w": 140, "h": 34, "cx": 180, "cy": 503}}
 ]}

Rects are viewport pixels, including DPI scale — the same space input.mouse --x --y takes, so you can click a button without a screenshot. cx/cy are the centre, because that is what every caller wants.

focusable and hasFocus are the two fields worth knowing about. UUserWidget defaults to not focusable, which makes SetFocus() a silent no-op and stops NativeOnKeyDown from ever firing — a screen that says "press any key" and then answers only the mouse. If you are debugging that, this verb tells you in one call.

There is no zOrder field. UMG takes z-order as an AddToViewport argument and exposes no runtime getter, so any number here would be invented.

log.tail

python aigauntlet.py log.tail --limit 20 --minVerbosity Warning

{"lines": [{"seq": 8801, "category": "LogAIGauntlet", "verbosity": "Display", "text": "Client authenticated."}],
 "nextCursor": 8802, "warnings": 3, "errors": 0, "dropped": 0}

Pass the previous reply's nextCursor back as since to read only what is new. Cursors are sequence numbers, not timestamps, so a polling loop never double-reads a line and never silently skips one.

dropped counts lines the 4096-entry ring overwrote between your calls. It is the only signal that your loop is too slow, so check it.

warnings and errors are counted over everything matching your filters, not over the page — "how many errors are there" should not change with limit.

input.text

python aigauntlet.py window.focus
python aigauntlet.py input.text --text "player one" --clearFirst true --commit true

clearFirst sends Ctrl+A so the typed text replaces what was there. commit appends Enter. perCharFrames spreads the characters over frames for a game that samples input once a tick; the default sends the whole string in one SendInput batch, so no other input can interleave.

os route only, and this is not a limitation waiting to be lifted. A text box is Slate, and the in-process routes do not reach Slate — they report success and deliver nothing. The verb refuses any other route rather than lying to you.

Gamepad

Eight pad.* verbs drive a simulated controller. They go in through the same FSlateApplication handler XInput uses, so UMG focus navigation sees the input and so does Enhanced Input.

g.pad_connect(layout="xbox")
g.pad_press("A")
g.pad_stick("left", 1.0, 0.0, frames=30)

Three things that will bite you, all covered in Gamepad.md:

• A stick latches. Without frames it stays deflected until pad.reset.
• X and Y are rejected unless the pad was connected as xbox — Xbox X is the left face button, PlayStation X is the bottom one.
• **A PlayStation D-pad is not Gamepad_DPad_*.** It is a HID hat on an analogue axis, and pad.connect has to arm it or your first press is swallowed.

VR

Two verbs, and the reason there are only two is worth knowing.

A VR controller's buttons and axes are ordinary FKeys. InputCore registers OculusTouch_*, ValveIndex_*, Vive_* and MixedReality_* itself - not an XR plugin, and not conditionally on hardware - so they exist with no headset attached, pad.button already accepts them, and the thumbstick axes carry Axis1D so GetInputAnalogKeyState reads them back. Measured on a packaged Shipping build with no VR hardware.

g.pad_press("OculusTouch_Right_A_Click")
g.call("input.axis", key="ValveIndex_Left_Thumbstick_X", delta=0.75, route="pc")

What is NOT a key is a pose, which is what vr.pose is for:

g.vr_pose("left", location={"X": 40, "Y": -25, "Z": 10}, rotation={"Pitch": 12})

part is hmd, left or right. Hands are matched on the motion controller's own motion source rather than on component name, because a name is whatever the game called it. Each block is optional and a partial write keeps what it omits.

Getting a VR pawn without a headset is the GAME's job, not the plugin's. A pawn class is the game's business and the plugin has no business knowing its class names. See vr_enter() in the client for the two reflected calls BACKLIT exposes; any game can expose the same pair.

transform.set

Each block is optional and a partial block keeps the components it omits, so you can nudge one axis without restating the other two. The reply carries the resulting transform, so you never need a read-back.

Registering your own

See CustomVerbs.md. Generic verbs let an agent do anything mechanically; semantic verbs make the transcript readable and the test short.

Protocol

Newline-delimited JSON, request and response, over a loopback TCP socket.

The handshake

The first line a client sends must be its token:

{"token":"3f1c...9ab2"}
{"ok":true,"result":{"authenticated":true}}

No verb runs before this succeeds, ping included. A wrong or missing token gets one error line and a closed socket:

{"ok":false,"error":"bad token - send {\"token\":\"...\"} first; the path is in the game log"}

The handshake reply carries no id — it is answered on the socket thread before any call object exists. A client that matches replies by id must read this one line separately. The bundled Python client does that for you.

Calls

{"id": 1, "verb": "ping", "args": {}}
{"id": 1, "ok": true, "result": {"map": "M1_Theatre", "frame": 4210}}

Responses are written as each call completes, not in arrival order, so a screenshot that takes three frames does not block a state query issued behind it. Match on id.

Verbs execute on the game thread, always. A verb may answer on a later frame — screenshot and input.text with perCharFrames both do.

Failures look like this, and the message is meant to be read by whoever caused it:

{"id": 7, "ok": false, "error": "asked for level 'Lamp' and no such prompt is loaded (60 are)"}

Object specs

Every verb that takes object accepts:

world  gameinstance  gamemode  gamestate  pc  pawn  playerstate  hud
actor:<Name>                  exact match, else prefix match
class:<ClassName>             first actor of that class
subsystem:<ClassName>         game instance or world subsystem
component:<Actor>/<CompName>  the owner is itself a spec, so component:gamemode/Round works
widget:<Name>                 a live UUserWidget, exact then prefix on name or class
/Game/Path/To.Asset           any object path
<BareName>                    falls back to an actor search

What comes back with every object

Any reply describing an object carries three underscore-prefixed fields you did not ask for, because they are what you need to address it again later:

• __name — the runtime name, which is what actor: and widget: match on.
• __class — the class name.
• __path — the full object path, which is the unambiguous way to address it.
• __transform — for actors and scene components, the live transform.

unknownProperties

When you pass properties[] and a name resolves to nothing, the reply says so:

{"__name": "BP_Prop_C_3", "Scale3D": {"X": 1, "Y": 1, "Z": 1},
 "unknownProperties": ["NotAPropertyAtAll"]}

This matters more than it looks. Before it existed, a bogus name was dropped silently, so "no such property" and "the value is empty" were indistinguishable — and a caller doing arithmetic on the missing number got a wrong answer with no error anywhere.

Scale3D, Scale, Location, Position, Rotation and Transform are aliases resolved against the live transform. None of them is a UPROPERTY on an actor (the transform lives on the root component, where the property is called RelativeScale3D), so without the alias table the obvious request returned nothing.

Input routes

There are four ways in and they are not interchangeable. This table was measured against a real packaged build, not inferred from engine source.

Use pc for gameplay

It is the same pipeline real hardware feeds, so triggers, modifiers and mapping contexts behave exactly as they do for a human.

That is also why tap defers its release by a couple of frames: a press and release in the same frame is invisible to any Enhanced Input Hold or Tap trigger. If you send your own down/up pair, leave frames between them.

Neither in-process route drives UI

A synthetic key through pc or slate, and a synthetic click through slate, all failed to dismiss a "press any key" title card across four measured attempts — including with the window forced to the foreground. Something in Slate's routing declines events that did not arrive through the platform message pump.

Two specific traps found while establishing that:

• Mouse buttons are not key events to Slate. LeftMouseButton through ProcessKeyDownEvent compiles, runs, reports success and delivers nothing. Pointer input needs ProcessMouseButtonDownEvent, and note the asymmetry — the up event takes no window argument.
• A widget that is not focusable never sees a key at all. UUserWidget defaults to not focusable, so SetFocus() is a silent no-op and neither NativeOnKeyDown nor NativeOnPreviewKeyDown can fire. The mouse works because pointer input needs no focus. widgets.find reports focusable for exactly this reason.

os is the route that works

SendInput posts to the OS input queue, so the engine cannot tell it from a real keyboard or mouse — window activation, focus, capture and message order all behave normally. It is the route that actually clicked through a main menu, a level roadmap and two tutorial pages with nobody at the keyboard.

Two things it needs:

• window.focus first. SetForegroundWindow alone is refused to a non-foreground process, so the plugin pairs it with AttachThreadInput.
• The machine, for the duration. os is genuinely global: it moves the real cursor and types into whatever window is focused. Do not run an os-route test while you have an editor window in front with unsaved work.

python aigauntlet.py window.focus
python aigauntlet.py input.mouse --x 130 --y 500 --route os
python aigauntlet.py input.key --key LeftMouseButton --event tap --route os

Better still, do not write coordinates at all — widgets.find returns rect centres:

from aigauntlet import Gauntlet
with Gauntlet() as g:
    g.click_widget("START")

The pad route

pad is the gamepad's own route, and the only one the pad.* verbs accept. It posts to FSlateApplication::OnControllerButtonPressed, OnControllerButtonReleased and OnControllerAnalog — the three calls XInputInterface makes once a frame for a physical pad — so it reaches both destinations for the same reason real hardware does: Slate and UMG see it first, and anything the UI does not consume flows on through the viewport to the player controller and Enhanced Input.

Unlike os, it needs no window focus and does not touch the machine: it is in-process, so a pad test can run while you work in another window.

There is deliberately no os route for a gamepad. Windows has no SendInput for a controller — a genuine virtual pad means a kernel driver such as ViGEmBus — so rather than offer a route that reports success and delivers nothing, the verbs refuse anything but pad.

See Gamepad.md, which has the part that is not obvious: a PlayStation pad outside Steam has no Gamepad_DPad_* at all.

The wheel

input.mouse --wheel N sends wheel notches, os route only. Worth knowing about because games bind real verbs to it — one binds depth to the wheel and resize to ctrl+wheel, which made two of its five core actions untestable until this existed.

Platforms

os verbs are Windows-only in this version and fail loudly elsewhere rather than reporting success and doing nothing. pc and slate are platform-neutral. The module declares PlatformAllowList: [Win64] accordingly.

If you are the one adding macOS or Linux

A macOS (CoreGraphics) and Linux (XTest) implementation was written and reviewed, then deliberately taken back out of the shipping module because there was no way to compile either one: development was on Windows, UE cannot cross-compile for macOS, and no Linux toolchain was available. Shipping a backend nobody has ever built is worse than shipping none — a buyer discovers it on their platform, not ours.

It lives in git history at commit 0a3ce51, and it is a reasonable starting point rather than a finished thing. What it already accounts for, and what you should keep:

• macOS gates event injection behind the Accessibility permission; until it is granted the route silently does nothing.
• macOS needs Command, not Control, for select-all, which is what input.text --clearFirst depends on.
• macOS virtual key codes are keyboard positions, not characters. Typed text should go through CGEventKeyboardSetUnicodeString, which is layout-independent.
• Linux XTest needs an X server; there is no Wayland equivalent without a portal handshake. Check XTestQueryExtension so an absent extension fails honestly.
• Linux typed text is practically ASCII-only: XTest needs a keycode, which exists only for a keysym the current layout carries, and arbitrary Unicode would mean rewriting the user's keyboard map — a global side effect on their session.
• X11 headers #define None, Bool, Status, Success, KeyPress and more, all of which collide with UE. An #undef block after the include is the standard price.
• Put the platform #includes above the namespace. Written inside it, they nest the whole of CoreGraphics or Xlib inside AIGauntlet::OsInput.
• Client-to-screen conversion is the open problem. Windows has ClientToScreen; the version in history goes through Slate's window position, which is exact for a borderless window and off by the decoration height for a titled one. Fix that properly before trusting a click position.

Gamepad simulation

Eight verbs: pad.connect, pad.disconnect, pad.button, pad.stick, pad.trigger, pad.dpad, pad.state, pad.reset.

from aigauntlet import Gauntlet
with Gauntlet() as g:
    g.pad_connect(layout="xbox")
    g.pad_press("A")
    g.pad_dpad("down")
    g.pad_stick("left", 1.0, 0.0, frames=30)

There are two kinds of pad on Windows, not one

That third row is the one that catches people. Outside Steam a Sony pad's D-pad is a single HID hat-switch usage (0x39), and RawInput can only surface a hat as an axis. There is no Gamepad_DPad_Up for it at all.

**So a simulator that sends only Gamepad_DPad_* tests the Xbox path and reports PlayStation coverage it does not have.** Any Sony-only D-pad defect stays invisible, and the test goes green. pad.dpad writes the hat under layout: "ps4" / "ps5", and every reply carries a path field saying which route it took.

The hat encoding

A DS4 hat declares logical range 0..7, and the axis arrives as Value / (LogicalMax - LogicalMin) — so raw/7.

The arming write, and why the first press would otherwise vanish

Raw 0 means "up". It is also what an analogue axis reads before its first HID report has ever arrived. A decoder cannot tell those apart, so a correctly written one carries an arming latch: it ignores the axis until it has seen the hat at rest at least once. Without that, every Sony pad would boot holding D-pad UP.

Against such a decoder, a simulator that opens with a direction has that direction discarded — and the run either reports a broken D-pad or, worse, quietly loses its first input and passes anyway.

So pad.connect on a PlayStation layout writes the neutral value once, immediately, and again after level travel, because a decoder living on a player controller is rebuilt by travel and re-arms from scratch. pad.connect's reply carries hatArmed so the transcript records it.

If the RawInput plugin is not enabled, GenericUSBController_Axis5 is not a registered key and pad.dpad fails naming that. It does not fall back to Gamepad_DPad_* — that fallback would turn "this project cannot test the Sony D-pad path" into "the Sony D-pad works".

🔴 On UE 5.8 the engine throws the hat away, and it does the same to a real DualShock

This is measured, and it is the single most important thing on this page.

RawInput registers GenericUSBController_Axis1..24 with FKeyDetails::GamepadKey and no Axis1D flag. FKey::IsAnalog() is therefore false for them. UPlayerInput::InputKey tests Params.Key.IsAnalog() to decide whether an event is analogue; a generic axis fails that test and takes the digital branch, whose switch (Params.Event) handles IE_Pressed, IE_Repeat, IE_Released and IE_DoubleClick — and has no IE_Axis case. Nothing is stored. GetInputAnalogKeyState reads 0 however hard you push.

The event is genuinely sent, genuinely reaches Slate, and genuinely reaches the viewport. The engine discards it at the last step.

Measured side by side, same call, same frame:

None of this is specific to the simulator. RawInput's own device code calls the same MessageHandler->OnControllerAnalog, so a physical DualShock on the raw HID path hits the identical wall. A game that reads its Sony D-pad through GetInputAnalogKeyState cannot work on this engine version — with a simulated pad or a real one.

pad.state reports hatReadableByGame, and pad.connect on a Sony layout returns a hatWarning saying exactly this, so a run cannot quietly conclude that the game ignores its D-pad.

The plugin does not re-register the key with Axis1D to work around it. EKeys::AddKey ensures on a duplicate, and a test harness that mutates its host's key table to make its own feature look better is worse than one that reports the truth.

The pad must be on Slate user 0

pad.connect therefore maps to GetPrimaryPlatformUser() — where a real first controller lands — and warns if the resulting Slate user index is not 0. Pass user: 1 deliberately if you want a second player.

This is also why the smoke suite asserts through the game's own GetInputAnalogKeyState rather than through pad.state. An entirely inert pad passes every shadow-state check there is.

A game still booting is not listening

Input during a startup movie or a loading screen reaches nothing, because focus is not on the game viewport yet — exactly as it is for a human mashing buttons over a logo. Wait for the route to be live before asserting on it:

for _ in range(40):
    g.pad_stick("left", 1.0, 0.0)
    g.call("wait", frames=5)
    live = abs(analog("Gamepad_LeftX")) > 1e-3
    g.pad_reset()
    if live:
        break

Measured on BACKLIT: six attempts. A suite that skips this reports a broken pad and means "you asked too early".

Buttons, and the two names that are rejected

Aliases are layout-scoped and case-insensitive. A raw UE FKey name always works.

Mixing dialects is also an error: Cross under an xbox layout fails, because a script that mixes them has lost track of which pad it is testing.

The default layout is neutral, which speaks both dialects for everything unambiguous — A and L1 both resolve — and rejects only X and Y. Its D-pad uses the Xbox path.

A raw FKey name is accepted only when the alias table has never heard of it, and only if the key is a gamepad key. A is a valid FKey — the keyboard A — so a fallback that fired on a rejected alias would press a keyboard key and look like it worked.

Sticks latch, and that is the trap

UPlayerInput::ProcessInputStack zeroes only RawValueAccumulator. RawValue survives a frame with no samples unless the key carries UpdateAxisWithoutSamples, and that flag is set on MouseX, MouseY, Mouse2D and MouseWheelAxis — and nothing else.

So a gamepad axis latches. Feed a stick once and it stays deflected until something feeds zero. That is correct, and it matches hardware: XInput reports a value only when it changes.

Four things guard it:

• pad.stick(..., frames=N) centres itself after N frames. Use this for a bounded gesture.
• pad.reset() centres and releases everything without disconnecting.
• pad.reset runs automatically on level travel and when the client's socket drops. A dropped connection cannot leave the game holding a stick.
• pad.state and the snapshot pad block report anythingHeld, and add a warning field when something is. If input is held, that is the first thing either reply says.

pad.stick also sends the four digital direction edges (Gamepad_LeftStick_Right and so on) alongside the analogue axes, as XInput does. Both are needed: UMG analogue navigation and a lot of game code read the digital directions, so a stick that sent only the axis would move a character and be unable to move a menu cursor.

deadzone: "xinput" snaps anything under XINPUT_GAMEPAD_LEFT_THUMB_DEADZONE (7849/32767 ≈ 0.2395) to exactly zero — what hardware would have sent. The default is none, so a test probing 0.1 gets 0.1.

🔴 Gamepad_RightY is negated, and it is the only key that is

FSceneViewport::OnAnalogValueChanged negates the analogue value for Gamepad_RightY specifically. Gamepad_LeftY is untouched. An identical "stick up" therefore reaches gameplay with one sign on the left stick and the opposite on the right.

This is engine behaviour applied equally to real hardware, so the simulator does not compensate — compensating would make a simulated pad behave differently from a physical one, which is the one thing it must never do.

--y is defined as the value the hardware reports: positive is stick-up on both sticks. pad.stick's reply adds yObservedByGameplay for the right stick, and pad.state lists the observed value beside the sent one, so the asymmetry is visible instead of remembered.

Triggers

pad.trigger sends the analogue axis and the threshold button, crossing at XInput's own threshold (30/255 ≈ 0.1176). An action bound to Gamepad_LeftTriggerAxis sees the first; an action bound to Gamepad_LeftTrigger as a key sees the second. Sending one without the other would leave half the bindings in a project untestable.

The route

pad is the only route these verbs accept. Input goes in through FSlateApplication::OnControllerButtonPressed, OnControllerButtonReleased and OnControllerAnalog — the three calls XInputInterface makes for a physical pad — so Slate and UMG see it first and anything the UI does not consume flows on through SViewport, FSceneViewport, UGameViewportClient, APlayerController and UPlayerInput to Enhanced Input. One route, both destinations. No driver, no administrator rights, and it works in Shipping.

There is no os route for a gamepad. Windows has no SendInput equivalent for a controller — a real virtual pad needs a kernel driver such as ViGEmBus — so rather than offer a route that reports success and delivers nothing, the verbs refuse anything but pad.

Driving a menu

import sys; sys.path.insert(0, "Plugins/AIGauntlet/Tools")
from aigauntlet import Gauntlet

with Gauntlet() as g:
    g.call("window.focus")
    g.pad_connect(layout="xbox")

    before = g.top_screen()
    g.pad_press("A")                 # dismiss whatever is in front
    g.call("wait", frames=20)

    g.pad_dpad("down"); g.call("wait", frames=10)
    g.pad_dpad("down"); g.call("wait", frames=10)
    g.pad_press("A");   g.call("wait", frames=30)

    print(before, "->", g.top_screen())
    print(g.call("screenshot", name="pad_menu")["path"])

Assert on top_screen() or widgets.find, never on the verbs returning ok. A verb that answers ok having delivered nothing is the failure this whole surface exists to avoid.

Security

AI Gauntlet is a remote-function-call surface. Anything it can reach, a caller can read, write or invoke.

Three things stand between that and a problem, and it is worth knowing exactly what each one covers: a checkbox (do not ship the plugin enabled), a launch flag (no -aigauntlet, no socket), and a token (no verb runs before a client proves it can read a user-scoped file). The checkbox is the one that matters for a release; the other two protect the build you forgot to untick.

The flag

No -aigauntlet on the command line, no socket. There is no config file, environment variable or ini key that opens it.

There used to be a bAutoStart ini fallback and it was removed, because a setting committed during a debugging session is precisely how a shipped build ends up listening. The flag is the only way in.

The token

On startup the game generates 256 bits of token, writes it to <ProjectSaved>/AIGauntlet/<pid>.token, and logs the path — never the token. (In a Shipping build the engine compiles that log call away, so find the file rather than the line; see Install.md for where a staged build puts it.) A client's first line must be {"token":"<the token>"}. Anything else gets one error line and a closed socket, so a guesser pays a fresh TCP connection per attempt. No verb runs before the handshake, ping included. The file is deleted on shutdown.

The token lives in a file rather than on the command line because a Windows process command line is readable by any other process through WMI. -aigtoken= would hand the secret to exactly the local process this is meant to stop.

The comparison is length-first and then a full-length scan with no early exit, so a wrong answer does not leak how many leading characters were right.

The Python client finds and sends the token automatically. In normal use you will not notice this exists.

The checkbox

The plugin compiles into every configuration, Shipping included, and there is exactly one control: the plugin's own Enabled tick. Disabled means the module is not compiled and not staged, so nothing of it reaches your game.

This replaced a compile-time gate, and the reason is worth knowing because it changes what you can rely on. A Fab code plugin arrives precompiled — Epic's build farm compiles it, not the buyer — so a compile-time switch is already resolved by the time the plugin lands, and a Blueprint-only project cannot rebuild it. The gate therefore meant "no Shipping support unless you can compile C++". One honest checkbox for everyone beats a guarantee that only holds for the minority who build from source.

🔴 Do not ship this plugin enabled to a store

Untick it before you build a release. That is the instruction, it is in the README and Install.md as well, and it is deliberately not dressed up as anything cleverer.

Two things make it checkable rather than a matter of memory: plugin enablement is recorded in your .uproject, so it appears in a diff; and a build with it disabled contains no AIGauntlet module, which you can confirm in the packaged Binaries folder.

If you build from source and want the old hard compile-out, AIGAUNTLET_ENABLED is defined to 1 by the plugin's Build.cs; set it to 0 in your own target rules and every part of the surface compiles to nothing. That is available, not default, and it does nothing for a precompiled install.

What this does not protect against

Stated plainly, because a security page that only lists strengths is not one.

• Another process running as you. A process with your user's rights can read the token file. The boundary here is the user account, not the process. If you do not trust code running as you, this plugin is not your problem.
• A build you deliberately shipped with the gate open and the flag passed. Nothing in the plugin can save a launcher script that hands -aigauntlet to players.
• Anything on the network. The listener binds 127.0.0.1 and the address is not configurable. Do not port-forward it. Do not tunnel it and call it a feature.
• Malicious arguments. A caller that authenticates can call any UFUNCTION in your game, including ones you would never expose deliberately, and write any UPROPERTY. That is the product, not a defect — treat authentication as full trust.
• Denial of service by a local caller. Verbs run on the game thread, capped at 64 per tick. A flood will make the game stutter.

If you are reviewing this for a studio

The three questions usually asked, answered directly:

1. Can a player reach it in a release build? Only if the plugin was left enabled when that build was compiled and the launcher passes -aigauntlet and the player can read the token file the game writes. All three, not any one. The intended answer is that the plugin is unticked for release, which is visible in the .uproject diff and belongs on your release checklist.
2. Is anything sent off the machine? No. There is no network code beyond a loopback listener, no telemetry, and no outbound connection of any kind.
3. What does it write to disk? One token file under Saved/AIGauntlet, deleted on shutdown, and screenshots you asked for.

Custom verbs

You may not need to. The generic verbs reach anything reflected, and one commercial game's entire round loop — start, judge, read the verdict, check the solve — was driven through nothing but object.call and object.get, with zero plugin-side game code. Check whether reflection already covers your API before writing C++.

Where custom verbs earn their place is readability. game.startRound --word broom says what a test is doing; three object.calls with a component path do not.

From C++

Register from your module's StartupModule, which runs long before any GameInstance exists — that is why the registry is static and independent of the subsystem.

#include "AIGauntletSubsystem.h"

FAIGauntletVerbRegistry::Get().Register(
    TEXT("game.startRound"),
    TEXT("Begin a round with a given word. args: word"),
    FAIGauntletVerb::CreateLambda([](const TSharedRef<FAIGauntletCall>& Call)
    {
        const FString Word = Call->GetString(TEXT("word"));
        if (Word.IsEmpty())
        {
            Call->Fail(TEXT("missing 'word'"));
            return;
        }

        // ... start the round ...

        const TSharedRef<FJsonObject> Result = MakeShared<FJsonObject>();
        Result->SetStringField(TEXT("word"), Word);
        Call->Complete(Result);
    }));

Your verb appears in describe automatically, so an agent discovers it without being told.

Rules

Verbs run on the game thread. Always. You do not need to marshal.

You do not have to answer this tick. Anything that needs the renderer, a frame of settling, or a network round trip holds its TSharedRef<FAIGauntletCall> and completes it later. Nothing blocks the game thread waiting.

// Answer in three frames.
TSharedRef<int32> Countdown = MakeShared<int32>(3);
FTSTicker::GetCoreTicker().AddTicker(FTickerDelegate::CreateLambda(
    [Call, Countdown](float) -> bool
    {
        if (--(*Countdown) > 0) { return true; }
        Call->CompleteEmpty();
        return false;
    }));

The registry keeps the last writer for a name, so registering input.key yourself overrides the built-in one. Useful for a project with an unusual input stack, and it avoids forking the plugin.

Fail with a sentence, not a code. The error string goes straight to whoever caused it — often a language model, which can act on asked for level 'Lamp' and no such prompt is loaded (60 are) and cannot act on E_INVALID_ARG.

Argument helpers

FAIGauntletCall carries the accessors every verb reaches for, so you are not parsing JSON by hand:

Call->GetString(TEXT("word"));
Call->GetNumber(TEXT("frames"), 3.0);
Call->GetBool(TEXT("visibleOnly"));
Call->HasField(TEXT("limit"));
Call->GetObject(TEXT("params"));
Call->GetArray(TEXT("tags"));
Call->GetVector(TEXT("location"), OutVector);   // reads {"X":..,"Y":..,"Z":..}
Call->GetRotator(TEXT("rotation"), OutRotator);

Gating

Your registration site should be guarded the same way the plugin is, or a Shipping build with the control surface compiled out will not link:

#if AIGAUNTLET_ENABLED
    FAIGauntletVerbRegistry::Get().Register(...);
#endif

The define is public, so your module sees it as long as you depend on AIGauntlet.

What the assistant reads (AGENTS.md)

You can inspect and control the running game over a local socket. The client is Plugins/AIGauntlet/Tools/aigauntlet.py, usable as a CLI or a library. The game must be running with -aigauntlet - and starting it is your job, not the user's.

Starting and stopping the game

python Plugins/AIGauntlet/Tools/aigauntlet.py launch                                   # editor in game mode
python Plugins/AIGauntlet/Tools/aigauntlet.py launch --exe Build\Windows\MyGame.exe    # a packaged build
python Plugins/AIGauntlet/Tools/aigauntlet.py launch --map MainMenu --res 1920x1080 --port 7799
python Plugins/AIGauntlet/Tools/aigauntlet.py stop

launch finds the .uproject above the working directory (or above the plugin), resolves the engine it is associated with, appends -aigauntlet -aigport -windowed -ResX -ResY, starts the process detached, and returns the ping reply once the socket answers - a cold editor start can take a minute or more, and it waits up to 240 s. It records the pid; stop ends that process and nothing else. If launch fails it says why: no project found, engine not installed on this machine (pass --exe), or the game exited before the socket opened, which almost always means the plugin is not enabled in the .uproject. Do not start the game by hand and do not kill it by image name; use these two.

If a game is already running with -aigauntlet, skip launch and just call verbs.

Which build am I driving?

Ask ping. The same verbs work in all of them, but two things change:

• An editor game-mode build (-game -aigauntlet) is Development: log.tail is rich, and a content change is a restart rather than a cook. Best for iterating.
• A packaged Shipping build has most UE_LOG calls compiled away, so log.tail is nearly empty there. Do not conclude the game is quiet - conclude the log is stripped. Read state and the screen instead. Measured on one game: 58 errors and 60 warnings in an editor build, 5 lines and nothing at Warning in Shipping.

Start here, always

python Plugins/AIGauntlet/Tools/aigauntlet.py orient

That prints the map, the framework classes with how many functions each exposes, what is on screen, the actor tag histogram, and whether the log is complaining. Read it before doing anything else — it is the difference between acting and guessing.

The loop

import sys
sys.path.insert(0, "Plugins/AIGauntlet/Tools")
from aigauntlet import Gauntlet

with Gauntlet() as g:
    print(g.orient())

    g.click_widget("START")          # finds the button by its label, clicks its centre
    g.call("wait", frames=10)

    print(g.snapshot()["map"])
    print(g.log_tail(limit=20)["lines"])
    print(g.call("screenshot", name="after_start")["path"])

Be fast: batch, click, wait

A round trip costs about 17 ms. Three verbs exist so you do not make hundreds of them.

## One round trip, not four.
g.batch(("ping", {}),
        ("object.get", {"object": "gamemode", "properties": ["__class"]}),
        ("wait", {"frames": 2}))

## One round trip, not six. Rects come from widgets.find.
g.click(244, 500)

## Waits on the GAME thread, not by polling from out here.
g.wait_until_state("component:gamemode/Round", function="IsRoundActive", equals="true")

Rules of thumb that matter more than they look:

• Never poll in a client loop when wait.until can do it. Its timeout also tells you what it last saw, which is usually the whole diagnosis.
• Never sleep to "let the UI settle". Wait for the state you actually want.
• Clicking is not input.key. Use input.click; tap's two-frame hold is right for an Enhanced Input trigger and too short for a Slate button.
• A batch child that takes frames is waited for before the next starts, so a whole input gesture fits in one batch. A batch cannot contain a batch.

Finding things

• Tags, not names. GetName() returns an internal name at runtime, so many games identify nothing by name. snapshot's tags histogram tells you which tags exist and how many actors carry each; then actors.find --tags '["ShelfProp"]'.
• components.find when you know the component class but not which actor owns it. This is common: a component created with NewObject at runtime hangs off whatever the game chose, which may be a coin flip between two candidates.
• widgets.find --text START for the UI. Rects come back in viewport pixels with a centre, so you can click without a screenshot.
• snapshot's callable lists are the game's own API. In a heavily-UFUNCTION-annotated project that is everything you need; call it with object.call.

Traps, in roughly the order you will hit them

• object.set on an array REPLACES the array. Writing ["Prop"] to an actor's Tags deletes every other tag it had. Read, modify, write. This has cost a whole test run.
• Neither in-process input route reaches the UI. route: "pc" drives gameplay through Enhanced Input; only route: "os" reaches Slate, because it goes through the platform message pump. Call window.focus first, and remember the os route moves the real mouse and types into the focused window — it owns the machine while it runs.
• A same-frame press and release is invisible to any Enhanced Input Hold or Tap trigger. Use event: "tap", which defers the release.
• A widget that reports focusable: false will never see a key. That is not your bug to work around; it is usually the game's bug to fix (SetIsFocusable(true) before SetFocus()).
• If the game defers a GPU readback, read twice. A verb that reports a value computed from a render target may return the PREVIOUS frame's answer. Call it, act, call it again. A whole search once plateaued because every step was scored against the pose before it.
• Check unknownProperties before trusting a number you did not see. A property name that does not exist is reported there rather than silently omitted.
• Watch dropped on log.tail. Non-zero means the ring buffer overwrote lines between your calls and you have a gap.
• A packaged build's token can be invisible to your Python. The Microsoft Store build - and any virtualenv built on it - runs in an AppContainer with a redirected LOCALAPPDATA, and that is exactly where a packaged build writes its token. PowerShell lists the file while Python reports the directory as absent. It reads as "the game is not running". The client now says so when it detects that interpreter; use a python.org install, or copy the token somewhere readable and pass --token.
• A gamepad axis LATCHES. Feed a stick once and it stays deflected until something feeds zero — that is what hardware does, since XInput only reports on change. Use pad_stick(..., frames=N), or pad_reset() when the gesture ends. pad_state()["pad"]["anythingHeld"] is the check, and it warns you itself.
• X and Y are rejected unless the pad was connected with layout="xbox". Xbox X is the LEFT face button and PlayStation X is the BOTTOM one, so the wrong layout presses a different button and the test still passes. Write Square or Cross.
• **A PlayStation D-pad is not Gamepad_DPad_*.** Outside Steam it is a HID hat switch on an analogue axis. Connect with layout="ps5" and pad.dpad uses it — including the arming write, without which a decoder's latch swallows your first press.
• A solve may auto-advance the game. If you inspect state after a success, you may be reading the next level. Capture what you need at the moment it happens.

What you cannot do

• Call a plain C++ function or read a plain member. Only UFUNCTIONs and UPROPERTYs exist to reflection. If something you need is not annotated, the way through is usually the player's own path — project a world position to screen, then click it.
• Screenshot under -nullrhi. There is no viewport. Run windowed and offscreen instead.
• Reach a widget's Slate internals. The UUserWidget is reachable; the SWidget behind it is not.
• Type into a field and be sure it landed, without checking. Read the field back with object.get on its widget: spec.

Changelog

1.3.1 — 2026-09-04

Store-review fixes. No verb changed.

Added

• The assistant path is now the documented front door. Docs/WithAnAssistant.md: enable, run with -aigauntlet, paste one paragraph into Claude Code, Cursor, Antigravity or Codex. README, Install and the listing lead with it; the by-hand reference follows. AGENTS.md joins CLAUDE.md at the plugin root as the same document under the name the other assistants look for, with Tools/sync_agent_docs.py --check to keep them identical. The in-editor tab opens on that page and has a Copy prompt for your assistant button.
• aigauntlet.py launch and stop. The assistant starts the game itself: launch finds the .uproject above the working directory or the plugin, resolves EngineAssociation through the Launcher's install list, the registry or a source-build GUID, starts the editor in game mode (or --exe for a packaged build) with -aigauntlet -aigport -windowed, and returns the ping reply when the socket answers - failing early if the process exits first, which is the "plugin not enabled" case. stop ends that pid and nothing else. --dry-run prints the command. So the buyer's whole setup is: enable the plugin, paste one paragraph.

• In-editor documentation. A new editor-only module, AIGauntletEditor, adds Tools > AI Gauntlet > AI Gauntlet Documentation (and a Window > Developer Tools entry): a tab listing every shipped page - README, Install, Verbs, Protocol, Input routes, Gamepad, Security, Custom verbs, CLAUDE.md, Changelog - with the text alongside, plus buttons to open the page externally, open the Tools folder and copy the quickstart command. It reads the Markdown files from the plugin folder at the moment it opens, so the in-editor text cannot drift from the files. The module is Type: Editor, so nothing of it is compiled or staged into a game.

Fixed

• Tools/build_pad_table_tests.bat is gone: Fab prohibits batch files inside a plugin. The same compile-and-run is now Tools/build_pad_table_tests.py, which finds Visual Studio through vswhere instead of a hard-coded path and deletes the .exe and .obj it produced, so nothing compiled is left in Tools/ for a reviewer to find.
• AIGauntlet.uplugin carries "EngineVersion": "5.8.0". Fab reads the supported engine version from the descriptor, not from the listing form.
• Config/FilterPlugin.ini added. Fab packages only Source/ and what this file lists, so without it a buyer received the module and none of README.md, CHANGELOG.md, CLAUDE.md, Docs/ or Tools/ - the Python client included.
• The listing's Technical details now name the documentation that ships in the folder, and a .gitignore keeps __pycache__/ and compiled test binaries out of the plugin tree.
• The runtime module now compiles on its own. RunUAT BuildPlugin - the same compile Fab's build farm runs - failed on six missing includes that the host game's shared PCH had been supplying: Tickable.h and UObject/WeakObjectPtr.h in the subsystem and server headers, Policies/CondensedJsonPrintPolicy.h, HAL/FileManager.h, the engine's Version.h and Engine/GameInstance.h. Every earlier build had been inside a game project, where nothing showed. Verified: BUILD SUCCESSFUL in a clean host project, both modules, packaged output carrying Docs/, Tools/, README, CHANGELOG and CLAUDE.md.

1.3.0 — 2026-08-27

Added

• vr.pose and vr.state — head and hand poses, and what the XR system and possessed pawn actually are. 32 verbs become 34.
• There is deliberately no vr.button. A VR controller's buttons and axes are ordinary FKeys: InputCore registers OculusTouch_*, ValveIndex_*, Vive_* and MixedReality_* itself, unconditionally, so they exist with no headset, pad.button already accepts them, and the axes carry Axis1D so GetInputAnalogKeyState reads them back. Measured on a packaged Shipping build with no VR hardware: eleven bindings across four runtimes pressed, held and released.
• Client helpers vr_enter(), vr_leave(), vr_pose(), vr_state().

Verified

• VR simulation on a packaged Shipping build with no VR hardware present. Eleven bindings across four runtimes - OculusTouch_*, ValveIndex_*, Vive_* and MixedReality_* - pressed, held and released through pad.button, each read back out of the game's own input state. The axes carry Axis1D, so GetInputAnalogKeyState returns them, which is what made a separate vr.button unnecessary.
• A written pose holds for as long as nothing is tracking: measured across 30 frames for both hands and the HMD. This was the design's one flagged unknown.
• Forcing a VR pawn was driven entirely through reflection against the host game's own UFUNCTIONs, so the plugin learned no class names and the game's production headset gate was never modified.
• ⚠ The suite that produced these numbers is coupled to the game it was written against - forcing a VR pawn is the game's job - so it is not shipped with the plugin. VR verification in your project is manual; the gamepad suite is the one you can run directly.

Known

• A written pose only holds while nothing is tracking. UMotionControllerComponent overwrites its transform from the XR system every frame; with no XR system there is nothing to overwrite it with. Measured: held across 30 frames. With a headset attached vr.pose returns a warning rather than pretending otherwise.
• Driving a VR pawn without a headset is a logic-only session — no stereo, no reprojection, no head motion. Forcing the pawn is the game's job, not the plugin's.

1.2.0 — 2026-08-26

Added

• Gamepad simulation — eight pad.* verbs: pad.connect, pad.disconnect, pad.button, pad.stick, pad.trigger, pad.dpad, pad.state, pad.reset. 24 verbs become 32. Input goes in through FSlateApplication::OnControllerButtonPressed and its two siblings — the three calls XInputInterface makes for a physical pad — so UMG focus navigation sees it first and anything the UI does not consume flows on to Enhanced Input. No driver, no administrator rights, works in Shipping.
• Both Windows pad paths, not one. A DualShock or DualSense outside Steam produces no Gamepad_DPad_* at all: its D-pad is a HID hat switch that RawInput can surface only as an analogue axis, with neutral at 8/7 — above full deflection. layout: "ps4" / "ps5" writes that encoding; layout: "xbox" writes Gamepad_DPad_*; both: true does both, which is the Steam Input case. When the hat axis is not a registered key the verb fails naming RawInput rather than falling back, because the fallback would turn "this project cannot test the Sony path" into "the Sony D-pad works".
• pad.connect arms the hat. Raw 0 is both "up" and "no report yet", so a decoder with an arming latch discards everything until it has seen the hat at rest. Without the arming write the first D-pad press of every PlayStation run vanishes and the test passes anyway.
• A layout-scoped alias table — A/Cross, LB/L1, Start/Options, and the rest. X and Y are rejected unless the pad was connected as xbox: Xbox X is the LEFT face button and PlayStation X is the BOTTOM one, so a script that writes "press X" against the wrong layout presses a different button and still passes its own assertion. A raw FKey name is accepted only when the alias table has never heard of it and the key is a gamepad key — A is a valid FKey, the keyboard A.
• pad.stick sends the four digital direction edges as well as the analogue axes, because UMG analogue navigation reads those — a stick that sent only the axis would move a character and be unable to move a menu cursor. pad.trigger likewise sends the threshold button as well as the axis.
• Client helpers: pad_connect(), pad_press(), pad_stick(), pad_trigger(), pad_dpad(), pad_state(), pad_reset().
• snapshot grows a pad block when a pad is connected.
• Docs/Gamepad.md, and a pad row in Docs/InputRoutes.md.

Fixed

• Nothing in the pad path can leave input held. A gamepad axis LATCHES — UPlayerInput zeroes only RawValueAccumulator, and UpdateAxisWithoutSamples is set on the four mouse keys and nothing else — so a stick nobody centred stays pushed for the rest of the run while every later reading is quietly wrong. The pad releases everything on pad.reset, on pad.disconnect, on level travel, and when the client's socket drops. pad.state and the snapshot block say so out loud when something is held.
• Every key is validated before it reaches Slate. Those handlers ensureMsgf on an invalid FKey rather than returning false, and FKey constructs from any FName, so a typo would have been a dialog and a callstack in the middle of an unattended run.

• The pad is mapped to the primary platform user, not a new one. GetPlatformUserForNewlyConnectedDevice() allocates a fresh platform user when the default one already owns a device — and it always does, because the keyboard is on it. A pad mapped that way lands on Slate user 1, which has no focused widget and no local player, so ProcessKeyDownEvent bubbles along an empty focus path and every button and axis goes nowhere. The verbs answered ok and pad.state reported the stick held while the game saw nothing. Found by reading the game's own GetInputAnalogKeyState instead of the plugin's shadow, which is now what the smoke suite does.

Verified

• 55/55 on Tools/pad_smoke.py against a packaged Shipping build, under both the xbox and the PlayStation layouts. The suite is game-agnostic and ships with the plugin.
• 53 table unit checks in Tools/test_pad_table.cpp, compiled with cl.exe alone - no engine, no editor, run by Tools/build_pad_table_tests.py - covering the alias table, the layout scoping and the hat encoding.
• A game driven from its main menu, through a level-select screen and into a live round with the pad as the only input. Its HUD prompt strip switched to controller glyphs on its own, which is the game noticing the pad rather than the plugin claiming it.
• The D-pad check discriminates the two paths rather than merely passing on both: the same preset cycled, and the game's hat edge counter moves by 2 under a PlayStation layout and by 0 under xbox.
• 🔴 Every check reads the value back out of the game's input state, never out of the plugin's own bookkeeping. That is not a stylistic preference: an entirely inert pad once passed 46 checks that all read the sender's shadow state, because the pad had been mapped to a Slate user with no focused widget and no local player. A simulation suite that asserts on what it sent proves nothing.

Known

• Gamepad_RightY is negated by FSceneViewport::OnAnalogValueChanged and no other key is. This is not compensated for — compensating would make a simulated pad behave differently from a physical one. pad.stick reports yObservedByGameplay instead.
• On UE 5.8 the engine discards the hat value, and does the same to a real DualShock. RawInput registers GenericUSBController_Axis1..24 with GamepadKey and no Axis1D, so FKey::IsAnalog() is false; UPlayerInput::InputKey takes the digital branch, whose switch has no IE_Axis case, and stores nothing. GetInputAnalogKeyState reads 0 however hard you push. Measured side by side on one frame: an Axis1D key sent identically read back 0.467, the generic axis read 0. The hat event really is sent and really reaches Slate and the viewport — the engine drops it at the last step, and RawInput's own device code goes through the same OnControllerAnalog, so physical hardware fares no better. A game that reads its Sony D-pad through GetInputAnalogKeyState cannot work on this engine version. pad.state reports hatReadableByGame and pad.connect returns a hatWarning. The plugin deliberately does not re-register the key with Axis1D: EKeys::AddKey ensures on a duplicate, and a harness that mutates its host's key table to flatter its own feature is worse than one that reports the truth.
• The PlayStation D-pad path needs the RawInput plugin, which UE 5.8 marks deprecated.
• Input during a startup movie or loading screen reaches nothing, because focus is not on the game viewport yet — the same for a human. Wait for the route to be live before asserting on it; pad_smoke.py shows the loop.

1.1.0 — 2026-08-26

Added

• snapshot — one call returning the map and viewport, the five framework objects with their BlueprintCallable signatures, the visible widget tree, an actor tag histogram, and recent warnings. Capped, and honest about what it clipped. Start here on a game you do not know.
• widgets.find — live UUserWidget enumeration with viewport-space rects (DPI scale included), rendered text pulled from the widget tree, and the focusable / hasFocus fields that explain a screen which answers the mouse but not the keyboard. A widget:<Name> object spec makes widgets reachable through object.get / object.set / object.call, which closes the one part of a UE game the plugin could not see.
• log.tail — cursor-paged reads of UE_LOG with category, substring and minimum-verbosity filters, plus warning and error counts and a dropped count for a polling loop that has fallen behind.
• input.text — types a whole string through KEYEVENTF_UNICODE, with clearFirst, commit, and an optional per-frame mode. OS route only, and it refuses the others rather than reporting success and delivering nothing.
• python aigauntlet.py orient — snapshot rendered as the paragraph you actually read.
• batch — run a whole sequence of verbs in ONE round trip, in order, on the game thread. A round trip costs about 17 ms on loopback and that floor is not negotiable; the number of them is. Measured: six reads took 99 ms separately and 18 ms batched. Children that do not answer this tick are waited for before the next starts, so an input gesture works inside a batch.
• input.click — a whole click in one verb: move, settle, press, hold, release. Six round trips became one, with a hold long enough for a Slate button that has just taken mouse capture and an optional activating click at a neutral point.
• wait.until — poll a UFUNCTION or property on the game thread until it matches, instead of paying a round trip per attempt. A timeout reports the last value it saw.
• Client helpers: widgets(), click_widget(text), click(), click_at(), click_label(), top_screen(), log_tail(), type_text(), snapshot(), orient(), batch(), wait_until_state().

Security

• A connection token is now required as the first line of every connection. Written to Saved/AIGauntlet/<pid>.token, logged by path only, deleted on shutdown, and found automatically by the client. See Docs/Security.md.
• One control: the plugin's own Enabled tick. The surface compiles into every configuration, Shipping included. Disabled means the module is not compiled and not staged, so nothing of it reaches your game. The instruction that goes with that is in the README, Install.md, Security.md and the plugin's own description in the editor: do not ship this plugin enabled in a build you send to a store.

This replaced a compile-time gate driven by a switch file, and the reason matters. A Fab code plugin arrives precompiled - Epic's build farm compiles it, not the buyer - so a compile-time decision is already baked when the plugin lands, and a Blueprint-only project has no way to rebuild it. The gate therefore meant "no Shipping support unless you can compile C++", which is most of the audience. One checkbox that behaves the same for everyone beats a guarantee that only holds for people who build from source.

Still available for teams who want it: AIGAUNTLET_ENABLED is defined to 1 by the plugin's Build.cs, and setting it to 0 in your own target rules compiles the whole surface away. Not the default, and it does nothing for a precompiled install.

• The bAutoStart ini fallback is removed. The command-line flag is the only way in.

Fixed

• object.get, actors.find and components.find reported nothing when asked for a property that does not exist, so "no such property" and "the value is empty" were indistinguishable — and a caller doing arithmetic on the missing number got a wrong answer with no error. Replies now carry unknownProperties.
• Scale3D, Scale, Location, Position, Rotation and Transform resolve as aliases against the live transform on actors and scene components. None of them is a UPROPERTY on an actor, so all of them used to return silence.

Docs

• Docs/ and Resources/ were empty. The README is now a landing page; install, protocol, the verb reference, the measured input-route table, the security model and custom-verb authoring each have their own page. CLAUDE.md ships inside the plugin so an agent learns the surface — and its traps — from the folder.

Verified

• 55/55 on the smoke suite against a packaged Shipping build (UE 5.8.1, Win64). Nine of those cover the speed verbs, including batch ordering, its bad-child reporting, its refusal to nest, and wait.until satisfying, timing out diagnostically and polling a property.

The rest covers the token handshake on raw sockets, widgets.find on a live title card, snapshot including the tag histogram, log.tail cursors, input.text, the property aliases, a scale round-trip, and a full round driven through reflection - travel to the map, begin, judge, end.

• A packaged Shipping build with the plugin enabled serves the full surface; the same build with no -aigauntlet on the command line opens no port at all. (An earlier compile-time switch was also verified in both directions before it was removed in favour of the single checkbox.)
• An editor game-mode build (UnrealEditor.exe <project> -game -aigauntlet) serves all 24 verbs against uncooked content - orient, snapshot and log.tail all driven there. Worth the note because it means no packaging step is needed for day-to-day use. It also measured the Shipping logging difference directly: the same log.tail call returned 58 errors and 60 warnings in the editor build against 5 lines and nothing at Warning in Shipping.
• Play-In-Editor is expected to work for the same reason (the listener is a UGameInstanceSubsystem, so it starts when a game instance does) but was not tested.
• input.text end to end: two player names typed into the party setup screen's real UEditableTextBox fields and read back off the widget tree. A whole PARTY round was then played through the game's own UI - menu, mode intro, setup, hold-to-peek, two secret builds submitted with ENTER, guesses typed into the real field, reveal, scoreboard - with no party function called through reflection.
• A campaign run reached 19 levels solved in sequence by following the results card's own NEXT button, at roughly 20 seconds a level.

Known limits

• log.tail is much thinner in a Shipping build, because the engine compiles most UE_LOG calls away there. Measured: five lines total, nothing at Warning or worse. The buffer, cursors and filters behave identically - there is simply less to read.
• A staged build's Saved directory is %LOCALAPPDATA%/<ProjectName>/Saved, not anywhere under the staging folder. The client searches both; a launcher that redirects %LOCALAPPDATA% needs AIGAUNTLET_TOKEN_DIR or --token.
• Windows only for the os route, which is the only route that reaches Slate UI. pc and slate are platform-neutral.
• widgets.find reports no zOrder: UMG exposes no runtime getter for it.
• A game that is force-killed or crashes leaves its token file behind, because the file is removed on clean shutdown. The client always picks the newest file, so this is harmless in practice; a stale file only matters if the newest one belongs to a process that has since died, in which case the handshake is refused and you delete the file.

1.0.0 — 2026-08-25

First release. 17 verbs, reflection-based inspection and invocation, three input routes, screenshots, and a Python client. Validated by driving a full commercial game's campaign end to end — 60 levels — and four side modes, through the socket alone, with no game-side plugin code.