Skip to content

Proposal: make Move drag atomic with an explicit start #326

Description

@zengfanfan

Problem

Move(drag=True) starts from the cursor's position at execution time. An agent that knows both endpoints must first move to the source and then issue a second tool call, leaving a race window in which the cursor can change. The current drag helper also does not expose a bounded duration for intermediate movement.

Proposed additive API

Extend the existing Move tool rather than add another tool:

Move(
    loc=[end_x, end_y],
    drag=True,
    from_loc=[start_x, start_y],  # optional; current cursor remains the default
    duration=0.25,               # optional; seconds, bounded to 0..10
)

Behavior and compatibility:

  • Omitting both new parameters preserves the existing current-cursor drag.
  • from_loc and duration are rejected unless drag=True.
  • Both endpoints are validated before any mouse input.
  • The mouse button is released in a finally path if pressing or movement raises.
  • A non-zero duration emits intermediate positions and always finishes at the requested endpoint.
  • Because Move can drag, its MCP annotations should describe it as destructive and non-idempotent.

Duplicate check

I searched open and closed issues/PRs for explicit-start, deterministic, duration-based, and mouse-drag proposals. I found no implementation other than my withdrawn earlier proposal (#312/#313). PR #93 proposes a multi-point MousePath, but it does not perform button-held drag-and-drop or make the existing Move drag atomic.

Validation prepared

  • Unit coverage for legacy compatibility, JSON-list coercion, endpoint validation, duration bounds, intermediate movement, and release-on-error.
  • Ruff lint/format checks on the changed files.
  • Manual Windows 10 drag smoke tests.

Design choices

  • Use additive from_loc and duration parameters on the existing Move tool.
  • Retain the current cursor as the default start for backward compatibility.
  • Correct the Move annotations in the same focused change because drag-and-drop is destructive and non-idempotent.

The complete implementation is available in #328. Feedback is welcome through PR review, but no reply to this issue is required before review or merge.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions