Skip to main content

Synthetic Mouse

Packet Utils owns the suite's synthetic mouse planner, recorder, lifecycle, and SDK action bridge. The feature is disabled by default under Synthetic Mouse in the Packet Utils configuration.

Configuration​

  • Use synthetic mouse routes supported SDK actions through cursor movement and a native AWT click rather than their normal reflection packet path.
  • Mouse humanization (enabled by default) applies deterministic account and logged-in-session fatigue modifiers to synthetic paths. It is independent of Use synthetic mouse and does not schedule gestures.
  • Mouse profile selects Default, Lazy, Fast, Recorded, or Trajectory motion. Recorded uses the most recently analyzed statistical profile. Trajectory retrieves physical gestures from the selected local dataset and uses the fallback order described below.
  • Show click locations and trails renders the validated target bounds, selected landing point, current planned path, traversed trail, and completed generated paths for 2.5 seconds. Green identifies trajectory generation, cyan identifies Recorded fallback, and white identifies Default, Fast, or Lazy generation. It is disabled by default and does not enable synthetic dispatch by itself.

The Mouse Options tab keeps recording state, profile isolation, dataset readiness, and dataset operations in the RuneLite sidebar. Its Open Trajectory Viewer action opens a lifecycle-managed Swing window with bucket counts, medians, coverage quality, and the latest physical and generated paths. Closing the window does not stop recording, and Packet Utils disposes it on shutdown. Clear removes the selected trajectory snapshot after confirmation and preserves the raw recording sessions.

Capture and local storage​

MouseRecorder captures physical AWT canvas events while recording is enabled. InputLockBypass marks synthetic events, so the recorder and trajectory extractor exclude generated movement. Capture does not inspect outgoing movement packets. RuneLite's native mouse recorder remains responsible for sampling, encoding, buffering, and transmitting the canvas events that synthetic playback produces.

The extractor segments movement before a press, wheel event, canvas exit, focus or view boundary, revision or canvas-mode change, and idle gaps longer than 250 ms. Press, drag, and release sequences produce DRAG gestures. Click-path retrieval uses MOVE_TO_CLICK; stored drags remain available for later planner specialization. The extractor rejects invalid timestamps, off-canvas samples, gestures outside 20 to 2,000 ms, endpoint displacement below 2 pixels, and paths with fewer than two positions.

Accepted gestures use a 50 ms AWT cadence model. The extractor interpolates coordinates by event timestamp and retains the exact first and last positions. For click gestures, the press coordinate and timestamp form the endpoint even when the recorder suppressed a nearby move under its sampling interval. This cadence describes normalized AWT observations, not packet capture.

Raw schema-2 JSONL sessions remain at:

~/.runelite/n3/mouse/recordings/<profile>/

The versioned trajectory snapshot remains local at:

~/.runelite/n3/mouse/trajectories/<profile>/trajectory-store-v1.json

Each stored point contains relative dx, dy, and offsetMillis values. The store excludes menu metadata. Stable content-derived IDs and persisted processed-ID metadata keep startup indexing and manual rebuilds idempotent even after reservoir eviction. Packet Utils writes complete temporary snapshots and replaces the prior store atomically.

The repository retains at most 10,000 trajectories across seven endpoint-distance buckets: [0,50), [50,100), [100,150), [150,200), [200,300), [300,500), and [500,infinity) pixels. The first six buckets retain 1,428 entries each; the final bucket retains 1,432. Persisted eligible-seen counters drive per-bucket reservoir sampling, which prevents a long-running profile from retaining only recent gestures. Invalid store records and corrupt JSONL rows are skipped. Raw sessions can rebuild the snapshot.

Trajectory generation​

TRAJECTORY builds an in-memory index by the seven distance buckets and 16 direction bins. A query visits the matching and adjacent distance buckets plus the target direction bin and its neighbors. It ranks at most five results with fixed endpoint-distance, direction, and mean-velocity weights.

The planner rotates and uniformly scales source paths onto the requested start and landing vector. This operation is endpoint-aligned affine warping. The planner does not perform dynamic time warping. It rejects scale outside 0.5 to 2.0 and rejects transformed paths that materially leave the live canvas.

Qualified generation selects two or three candidates, resamples them by normalized arc length, assigns 60 percent weight to the highest-ranked candidate, and blends the remaining weight by score. The highest-ranked path supplies normalized temporal progress, preserving its acceleration shape when the planner retimes the blend. Learned per-bucket duration, point-count, perpendicular residual, and lag-one correlation statistics control output. Noise affects interior points only. Start and landing coordinates remain exact.

The dataset needs at least 20 valid click trajectories and three indexed candidates in the relevant or adjacent buckets. Normal mode uses learned medians. ActionPacer.isRapidActionMode() remains the rapid-state owner, and trajectory rapid mode uses the learned 25th-percentile duration and point count. Generated duration stays within 50 to 800 ms, point count stays within 2 to 64, residual sigma stays within 0 to 3 pixels, and correlation stays within 0 to 0.95.

Trajectory requests use this planner order:

  1. Qualified trajectory retrieval and generation.
  2. The selected Recorded statistical profile with MousePathPlanner.
  3. The standard MousePathPlanner profile.

An empty, corrupt, rebuilding, or underqualified dataset does not reject an otherwise valid click. Planner fallback stays inside the AWT transport. Synthetic dispatch never changes to reflection packets after the caller requested synthetic input.

Supported actions​

The runtime bridge currently covers:

  • NPCActions.interact(...)
  • ObjectActions.interact(...)
  • paced WidgetActions operations without a composite resume-count step
  • UseItemActions.dragAndDrop(...), which uses a held-button AWT drag only while synthetic mode is requested
  • Walker land movement, including the ground movement used after an imported adjacent-barrier transition

The shared dispatcher also exposes validated native scrolling for SDK consumers. It is not attached to a gameplay action or an Agent Server write endpoint.

Other action classes retain their existing dispatch mechanisms. Enabling the toggle does not replace every interaction suite-wide. Walker doors, NPCs, objects, widgets, teleports, and transports remain on their established Api.actions paths.

WidgetActions.interactImmediate(...) is outside the current synthetic bridge. Power Skiller uses that method for each operation in a drop burst, so enabling Use synthetic mouse does not reroute those operations or apply the headless-mode synthetic rejection. Planned synthetic handling for burst operations will use the existing rapid-action mouse-profile determination; the current implementation sends a direct mouse packet and invokes the native menu dispatcher.

When synthetic dispatch is disabled, direct packet actions still resolve a truncated, center-biased Gaussian landing point inside the target's live canvas shape before sending EventMouseClick. This selects only a landing coordinate; recorded motion profiles remain exclusive to cursor-path planning. A null, empty, off-screen, or otherwise unprojectable shape fails closed and suppresses the action packet rather than falling back to 0,0. Deprecated parameterless packet helpers select a point across the live canvas for compatibility, but target-aware callers must use the shape overload.

The overlay reads only snapshots produced by SyntheticMouseService. Packet-only operations have no synthetic target or route and therefore produce no marker.

For supported actions, the dispatcher validates the requested operation, selects a landing point inside the live clickbox, and starts the shared EDT timer. It accepts only one path owner at a time. It calls ActionPacer.recordAction() as soon as it accepts the request so callers yield while movement is in progress. For NPC and Object targets, action resolution maps directly to the entity's raw composition action slot (1-based), preserving null slots so entities with non-contiguous action definitions (such as Gemstone Crabs with "Attack" at index 1 or shed shells with "Mine" at index 2) retain exact menu entry construction and validation.

On arrival, the EDT hands off to RuneLite's client thread. That thread reads the current menu, installs the retained target as the forced left-click entry, issues the native click, and restores the preceding menu even when dispatch fails. The canvas receives MOUSE_MOVED, MOUSE_PRESSED, MOUSE_RELEASED, and MOUSE_CLICKED; the client then restores the prior menu and focusability state. A movement-dispatch failure suppresses the arrival click. A synthetic drag instead moves to a contained source point, sends MOUSE_PRESSED, sends paced MOUSE_DRAGGED events with the left-button mask to a contained destination point, and releases without a MOUSE_CLICKED event. Scroll uses MouseWheelEvent at the selected in-canvas point.

Walker selects the furthest door-safe tile in its normal randomized lookahead, then tries nearer tiles when the preferred tile has no live canvas polygon. A queued cursor path is only an attempt: Walker waits while it is pending and advances only after it observes player movement, route progress, or arrival.

Account and session modifiers​

When Mouse humanization is enabled, Packet Utils binds a deterministic profile after login. The profile includes speed, acceleration, curvature, jitter, staged-correction probabilities, camera mode and frequency, preferred glance tab, micro-break frequency, reaction-delay mean and standard deviation, and fatigue rate. A bounded six-hour logged-in-session fatigue factor further reduces speed and increases jitter. Logout pauses the session and clears the account profile; account changes reset it.

The Mouse Behavior diagnostics panel shows those account values and offers manual arrow-camera, middle-drag camera, XP glance, Equipment glance, predictive hover, mouse nudge, and offscreen-rest checks. Held arrow and middle-button input is released on its scheduled game tick and during disable, logout, account change, or shutdown cleanup. Manual synthetic paths use their own owner and never click; tab glances restore the previous tab only while the requested glance tab remains open.

Cancellation and failure behavior​

Manual canvas input, logout, plugin shutdown, canvas-size changes, or view-mode changes cancel the current path. Requests fail closed when the mouse is busy or a target has no usable clickbox. They do not silently fall back to reflection packets. In particular, once synthetic mode is requested, UseItemActions.dragAndDrop(...) never falls back to its widget-drag packet after an unavailable or cancelled synthetic request. Walker applies the same strict fail-closed rule: it retries busy or unprojectable ground movement through its bounded stall/replan policy and never sends a direct movement packet while synthetic mode is enabled. Cancellation, replacement, replan, logout, and Packet Utils shutdown invalidate request-correlated arrivals so a stale callback cannot click later.

Headless mode rejects every synthetic AWT profile, including Trajectory, with synthetic_disabled_in_headless_mode. Dataset availability does not weaken that boundary.

Input backend boundary​

The runtime reports the current backend as AWT_CANVAS_ONLY. It dispatches events to the RuneLite canvas and does not move the operating-system pointer. Agent Server capability output exposes both the backend and osPointerSynchronized=false. A workflow that requires pointer parity must run in an isolated display with an appropriate native-input owner; the plugin does not move a user's desktop pointer with Robot.

Focus boundary​

The click helper temporarily disables canvas focusability only when the canvas is not already the focus owner. This prevents the synthetic press handler from stealing OS keyboard focus and restores the prior focusable state afterward.

This feature does not alter the client's real focused, minimized, resized, or occluded state and does not spoof focus/window telemetry. That behavior requires separate, revision-sensitive injected-client ownership and is outside this external-plugin implementation.

Verification boundary​

Unit tests cover extraction boundaries, 50 ms interpolation, stable IDs, atomic persistence, profile isolation, stratified reservoir limits, indexed retrieval, deterministic affine blending, exact endpoints, timing preservation, learned rapid behavior, planner fallback, client-thread arrival queueing, exclusivity, cancellation, event order, focusability restoration, suppression of clicks after movement failure, bounded immutable histories, source colors, target containment, and sparse composition action array indexing for NPC and Object synthetic dispatch. Walker tests cover legacy packet ordering, nearer-projectable fallback, pending-state waiting, observed progress, and correlated cancellation. Static tests cannot prove live menu resolution, stretched-canvas targeting, OS focus behavior, keyboard focus preservation, or current-revision network behavior.

Live client verification: pending