How to Teach Claude Code to Control an iOS Simulator

Imagine: you're sitting at the terminal, chatting with Claude Code, and instead of manually opening Xcode, hunting for the right button in the simulator, or copying logs from the console, you simply write: "Check how the login screen looks on iPhone 15 and tap the sign-in button." And the AI does it. Sounds like magic, but in reality it's the work of the ios-simulator-skill script collection.

I stumbled upon this repository when looking for a way to integrate agentic AI with mobile development. The project author, Conor Ladhy, put together 22 Python and Bash scripts that turn Claude into a full-fledged QA engineer or development assistant capable of "seeing" an app's interface and interacting with it.

Why Bother If You Have Xcode

Usually, an AI agent's interaction with an iOS project turns into a nightmare due to massive build logs and heavy screenshots. If you feed Claude the standard output xcodebuild, the context window gets flooded with junk in a couple of seconds.

This project solves the problem elegantly. The scripts act as a filter: they feed the neural network only the essentials. Instead of 400 lines of authentication logs, Claude gets 15 lines of clean text. This saves tokens and, more importantly, prevents the model from "hallucinating" due to information overload.

What This Toolset Can Do

The project addresses two major tasks: building the app and manipulating the simulator.

Smart Build Without the Noise

The build_and_test.py script wraps the standard Apple tools. When you run a build, Claude sees only a short status line and a result ID. If something goes wrong, the model can specifically request errors or warnings for that ID. This is called progressive data disclosure — we don't dump everything at once, but provide details on demand.

Navigation via Accessibility

This is probably the coolest feature. Instead of trying to guess button coordinates on a screenshot (which often leads to misses), the scripts use the iOS Accessibility API. Claude searches for elements by their semantic meaning. For example, a command to find the text "Login" and click on it works more reliably than clicking at coordinates x: 320, y: 400. Plus, a textual description of the element tree takes hundreds of times fewer tokens than an image.

Screenshot Optimization

If you can't do without visual verification (for example, you need to compare design or find a layout bug), the scripts automatically compress and resize screenshots. This reduces context load by 90-95%.

How It Works Under the Hood

The repository contains a bunch of Python scripts that call xcrun simctl and idb (a Facebook tool for iOS automation).

Here are a couple of interesting tools from the list:

• screen_mapper.py — analyzes the screen and compiles a list of all interactive elements.
• gesture.py — can perform swipes, pinches, and even pull-to-refresh.
• privacy_manager.py — lets you grant an app permissions for photos, contacts, or location with a single command. This is incredibly convenient for automated tests where you previously had to manually click through system alerts.

Interestingly, the author even added accessibility_audit.py for checking app compliance with WCAG standards. Claude can walk through screens and tell you where you forgot to add labels for VoiceOver.

How to Run It Yourself

The project is positioned as a "skill" for Claude Code, but in fact it's a set of standalone scripts that can also be used separately.

If you're using Claude Code, installation looks like this:

/plugin marketplace add conorluddy/ios-simulator-skill
/plugin install ios-simulator-skill@conorluddy

The dependencies you'll need are Xcode Command Line Tools, Python 3, and preferably idb (installed via brew). If you plan to compare screenshots, also install the Pillow library via pip.

Practical Benefits

Who is this useful for? First and foremost, those who want to automate routine tasks. For example, you can ask Claude to:

1. "Build the project, launch it on iPhone 14 Pro, and take screenshots of all screens for documentation."
2. "Find why the test crashes at the authorization stage and show me logs from the last 2 minutes only."
3. "Check how the app behaves with slow internet and disabled geolocation."

In the author's own tests, the Claude + ios-simulator-skill combo showed 100% scenario completion, while "bare" Claude without these tools succeeded less than half the time.

Is It Worth Trying

If you're an iOS developer and have already started integrating AI agents into your workflow, then this project is a must-have. It removes the main pain point: the incompatibility of verbose Apple tools with the conciseness that language models need.

Of course, the project requires environment setup, and idb can sometimes be finicky on new macOS versions, but the idea of controlling a simulator through a semantic element tree is exactly what modern automation should look like. The scripts are written simply, they're easy to tweak for your needs, or you can extract just what you need for your CI/CD pipelines.