# Strategies and self-healing

> Learn how the StrategyManager tries each way to find an element in turn, and how a keyword falls back to the next result when its action fails.

Optics uses the Strategy pattern to find elements in more than one way. This page tells you how the strategies work, how Optics selects them, and how the fallback mechanism keeps a test running when one method fails.

This is the internal view. For the user view, see [Define elements](https://optics-framework.org/docs/elements/define), [Text and images](https://optics-framework.org/docs/elements/text-and-images), [Area of interest](https://optics-framework.org/docs/elements/area-of-interest) and [AI self-heal](https://optics-framework.org/docs/elements/self-heal).

## Overview

A strategy is one method to find an element. Optics tries the strategies in a fixed order. When one strategy fails, it tries the next. The result is a self-healing system: a test continues when the UI changes or the application is in a different state.

AI self-heal is a separate, last step. It runs only when every strategy fails. See [AI self-heal](https://optics-framework.org/docs/elements/self-heal).

## Strategy architecture

`How a keyword reaches the locator strategies`:

```mermaid
graph TB
    A[ActionKeyword] --> B[StrategyManager]
    B --> C[StrategyFactory]
    C --> D[XPathStrategy]
    C --> E[TextElementStrategy]
    C --> F[TextDetectionStrategy]
    C --> G[ImageDetectionStrategy]
    D --> H[ElementSource]
    E --> H
    F --> I[TextDetection]
    F --> H
    G --> J[ImageDetection]
    G --> H
    B --> K[Fallback Chain]
    K --> L[Try Next Strategy]
```

## StrategyManager

**Location:** `optics_framework/common/strategies.py`

The `StrategyManager` controls the location strategies and the fallback chain. `ActionKeyword`, `Verifier`, the app management keywords, the test runner and the error detection in `Optics` each make one.

### Key responsibilities

- Strategy creation and management
- Strategy execution with fallback
- Screenshot and page source strategy management
- Element location coordination
- Assertion polling for `assert_presence` and `assert_visibility`
- The list of interactive elements on the screen (`get_interactive_elements`)

### Initialization

```python
strategy_manager = StrategyManager(
    element_source=element_source_fallback,
    text_detection=text_detection_fallback,
    image_detection=image_detection_fallback
)
```

The full signature has one more argument:

```python
class StrategyManager:
    def __init__(
        self, element_source: InstanceFallback[ElementSourceInterface], text_detection, image_detection,
        annotate_assertions: bool = True,
    ):
```

- If `element_source` is not an `InstanceFallback`, the manager wraps it in one.
- `annotate_assertions` controls whether a passed assertion takes a screenshot to box the elements it found. The app management keywords set it to `False`.

### Strategy building

The manager builds four lists:

1. **Locator strategies** (`locator_strategies`): for element location. `StrategyFactory` makes them.
2. **Screenshot strategies** (`screenshot_strategies`): for screen capture. `ScreenshotFactory` makes them.
3. **Page source strategies** (`pagesource_strategies`): for page source extraction. `PagesourceFactory` makes them.
4. **Screen element supplements** (`screen_element_supplements`): sources such as `appium_web` that set `SUPPLEMENTS_SCREEN_ELEMENTS = True`. Their interactive elements are merged into the list from the native source.

The manager builds the locator strategies one element source at a time, in the order of `elements_sources` in config.yaml. Inside one source, the strategies are sorted by priority. The priority does not sort strategies across sources. For example, with `appium_find_element` before `appium_screenshot`, the order is:

1. `XPathStrategy` and `TextElementStrategy` on `appium_find_element`
2. `TextDetectionStrategy` and `ImageDetectionStrategy` on `appium_screenshot`

## Locator strategies

Every locator strategy extends the abstract class `LocatorStrategy`. It must give:

- an `element_source` property
- `locate(element, index=0)`: an element object, `(x, y)` coordinates, a `LocateValueWithFrame` (vision strategies), or `None`
- `check_elements(elements, visible, frame)`: one look per element with no wait, for assertion polling
- a static `supports(element_type, element_source)`

`can_assert(element)` is optional. It tells the manager whether the strategy can look up that element. The default is `True`.

### XPathStrategy

**Priority:** 1 (Highest)

Finds elements with XPath expressions through the element source.

#### When used

- Element type is "XPath"
- Element source supports `locate()` method
- Fastest method when available

#### Implementation

```python
def locate(self, element: str, index: int = 0):
    return self.element_source.locate(element, index)
```

#### Examples

**Basic XPath:**

```python
# Element: "//button[@id='submit']"
# Strategy: XPathStrategy
# Process: Direct element location via XPath
# Result: Element object or coordinates
press_element("//button[@id='submit']")
```

**XPath with attributes:**

```python
# Complex XPath with multiple attributes
press_element("//android.widget.Button[@resource-id='com.app:id/button' and @text='Submit']")
```

**XPath with index:**

```python
# Get second matching element
press_element("//button[@class='action']", index=1)
```

**XPath performance:**

- Fastest strategy (typically < 0.1s)
- Direct DOM access
- No screenshot required
- No vision processing

### TextElementStrategy

**Priority:** 2

Finds elements by their text directly through the element source.

#### When used

- Element type is "Text", "CSS" or "Class"
- Element source supports direct text location
- Faster than OCR-based detection
- Not used when the element starts with `text_only:`. That prefix sends the element to OCR only.

#### Implementation

```python
def locate(self, element: str, index: int = 0):
    return self.element_source.locate(element, index)
```

#### Examples

**Text-based location:**

```python
# Element: "Submit Button"
# Strategy: TextElementStrategy
# Process: Direct text-based element location
# Result: Element object
press_element("Submit Button")
```

**CSS selector:**

```python
# CSS selector (if supported by driver)
press_element("button.submit-btn")
```

**ID-based:**

```python
# Element ID
press_element("submit-button-id")
```

A bare id has no prefix, so its element type is "Text". The element source decides how to match it. For example, `selenium_find_element` tries visible text first, then id, name and link text.

**Performance:**

- Fast strategy (typically < 0.2s)
- Direct element access
- No screenshot required
- Driver-specific implementation

### TextDetectionStrategy

**Priority:** 3

Finds text elements with OCR (Optical Character Recognition).

#### When used

- Element type is "Text"
- XPath and direct text location failed
- OCR model is available
- Element source supports screenshot capture
- The element is not a CSS selector (`can_assert` refuses a `css=` element)

#### Implementation flow

`How TextDetectionStrategy finds text with OCR`:

```mermaid
sequenceDiagram
    participant TS as TextDetectionStrategy
    participant ES as ElementSource
    participant OCR as TextDetection
    participant SM as StrategyManager

    TS->>ES: capture()
    ES-->>TS: screenshot
    TS->>OCR: detect_text(screenshot)
    OCR-->>TS: text locations
    TS->>TS: find matching text
    TS-->>SM: coordinates
```

The strategy calls `text_detection.find_element(screenshot, text, index=index, strict=...)`. `TextInterface.find_element` calls `detect_text` and then matches the text. It then maps the point from screenshot pixels to window points (`utils.screenshot_point_to_window`).

#### Features

- Screenshot-based detection
- Full text detection with bounding boxes
- Exact matching of complete words, in order, the same for every OCR engine (`optics_framework/common/text_match.py`). Confidence is not used to match. `strict_element_match` in config.yaml makes the match strict.
- One look per call. The wait for an element to appear comes from assertion polling and from the keywords that retry, not from the strategy.

#### Examples

**Basic text detection:**

```python
# Element: "Login"
# Strategy: TextDetectionStrategy
# Process:
#   1. Capture screenshot
#   2. Run OCR on screenshot
#   3. Find "Login" text in OCR results
#   4. Return center coordinates
press_element("Login")
```

**Text with index:**

```python
# Get second occurrence of "Submit"
press_element("Submit", index=1)
```

**Text in a specific language:**

Configure the OCR engine for the language. `easyocr` reads English only from config.yaml. `apple_vision` takes a `languages` list:

`config.yaml`:

```yaml
text_detection:
  - apple_vision:
      enabled: true
      capabilities:
        languages: ["en-US", "es-ES"]
```

Then use the text in the test:

```python
press_element("Iniciar Sesión")  # Spanish text
```

**Performance:**

- Slower strategy (typically 1-3s)
- Requires screenshot capture
- OCR processing time
- Accuracy depends on image quality

### ImageDetectionStrategy

**Priority:** 4 (Lowest)

Finds elements with image template matching.

#### When used

- Element type is "Image"
- Other strategies failed
- Image detection model is available
- Element source supports screenshot capture

#### Implementation flow

`How ImageDetectionStrategy matches a template`:

```mermaid
sequenceDiagram
    participant IDS as ImageDetectionStrategy
    participant ES as ElementSource
    participant IM as ImageDetection
    participant SM as StrategyManager

    IDS->>ES: capture()
    ES-->>IDS: screenshot
    IDS->>IM: find_element(screenshot, template)
    IM-->>IDS: coordinates
    IDS-->>SM: coordinates
```

#### Features

- Template-based matching across screen sizes (scale sweep)
- SIFT/FLANN feature matching as the last fallback
- Area of Interest (AOI) support
- Multiple match support (index-based)

#### AOI support

The strategy can find images inside an Area of Interest (AOI). An AOI limits the search to one region of the screen. This makes the search faster and more accurate.

**What is AOI?**
An Area of Interest (AOI) is a region of the screen, given in percentages. Optics does not search the full screen. It crops the screenshot to the AOI, searches in the crop, then moves the coordinates back to the full screen.

**Why use AOI?**

- **Performance**: Smaller search area = faster location
- **Accuracy**: Reduces false positives from similar elements elsewhere
- **Efficiency**: Focuses vision models on relevant screen area
- **Flexibility**: Works with any screen size (percentage-based)

**AOI parameters:**

- `aoi_x`: X percentage of AOI top-left corner (0-100)
- `aoi_y`: Y percentage of AOI top-left corner (0-100)
- `aoi_width`: Width percentage of AOI (0-100)
- `aoi_height`: Height percentage of AOI (0-100)

`StrategyManager` requires all four together. If you give some and not all, it raises `E0205`.

**Example:**

```python
# Locate element in top-right quarter of screen
press_element(
    "submit_button",
    aoi_x="50",      # Start at 50% from left
    aoi_y="0",       # Start at top
    aoi_width="50",  # 50% width
    aoi_height="50"  # 50% height
)
```

**AOI implementation flow:**

`How a strategy searches inside an area of interest`:

```mermaid
sequenceDiagram
    participant Action
    participant Strategy
    participant Utils
    participant Vision

    Action->>Strategy: locate_with_aoi(element, aoi_x, aoi_y, aoi_width, aoi_height)
    Strategy->>Strategy: Capture full screenshot
    Strategy->>Utils: crop_screenshot_to_aoi(screenshot, aoi_x, aoi_y, aoi_width, aoi_height)
    Utils->>Utils: Calculate pixel bounds from percentages
    Utils->>Utils: Crop screenshot to bounds
    Utils-->>Strategy: cropped_screenshot, aoi_bounds
    Strategy->>Vision: find_element(cropped_screenshot, template)
    Vision-->>Strategy: coordinates (relative to cropped)
    Strategy->>Utils: adjust_coordinates_for_aoi(coordinates, aoi_bounds)
    Utils-->>Strategy: coordinates (relative to full screenshot)
    Strategy-->>Action: Final coordinates
```

If the crop or the adjustment fails, the strategy raises `E0205`.

**Coordinate adjustment** (`optics_framework/common/utils.py`):

```python
def adjust_coordinates_for_aoi(coordinates, aoi_bounds):
    """
    Adjust coordinates found in cropped AOI back to full screenshot.

    Args:
        coordinates: (x, y) found in cropped screenshot
        aoi_bounds: (x1, y1, x2, y2) pixel bounds of AOI

    Returns:
        (adjusted_x, adjusted_y) in full screenshot coordinates
    """
    x, y = coordinates
    x1, y1, x2, y2 = aoi_bounds
    adjusted_x = x1 + x
    adjusted_y = y1 + y
    return (adjusted_x, adjusted_y)
```

The real function also checks that `coordinates` is an `(x, y)` pair and `aoi_bounds` has four numbers. Otherwise it raises `ValueError`.

**AOI validation:**

```python
def calculate_aoi_bounds(screenshot_shape, aoi_x, aoi_y, aoi_width, aoi_height):
    """
    Calculate pixel bounds for AOI from percentage coordinates.

    Validates:
    - All parameters between 0-100
    - AOI doesn't exceed screen bounds
    - Width and height > 0
    """
    # Validation logic
    # Calculate pixel bounds
    return (x1, y1, x2, y2)
```

It also requires all four values to be numbers. It raises `ValueError` when `aoi_x + aoi_width` or `aoi_y + aoi_height` is more than 100.

**AOI annotation:**
When you use an AOI, Optics draws the AOI on the saved screenshot, for debugging:

```python
def annotate_aoi_region(screenshot, aoi_x, aoi_y, aoi_width, aoi_height):
    """
    Draw rectangle on screenshot showing AOI region.

    - Blue rectangle shows AOI bounds
    - Label shows AOI parameters
    - Saved to execution output directory
    """
```

The colour is `(255, 0, 0)` in BGR, which is blue. The label reads `AOI: x%,y% (w%xh%)`. If the bounds are not valid, the function returns the screenshot without a rectangle.

**Supported strategies:**

- `ImageDetectionStrategy`: Supports `locate_with_aoi()`
- `TextDetectionStrategy`: Supports `locate_with_aoi()`
- `XPathStrategy`: AOI not applicable (uses DOM structure)
- `TextElementStrategy`: AOI not applicable (uses DOM structure)

`XPathStrategy` and `TextElementStrategy` cannot crop, so `StrategyManager` filters their results instead. It finds the centre of the element (or the point) and keeps the match only if the centre is inside the AOI. A match outside the AOI is logged as a failed attempt, and the next strategy gets a try. This is a centre-point test, so at the edge of the AOI it can disagree with an OCR or image crop.

**Use cases:**

1. **Button in specific region**: Locate button only in bottom navigation
2. **Form fields**: Search only in form area, not entire screen
3. **Modal dialogs**: Focus search on dialog region
4. **Performance optimization**: Reduce search area for faster location

#### Examples

**Basic image matching:**

```python
# Element: "button_template.png"
# Strategy: ImageDetectionStrategy
# Process:
#   1. Capture screenshot
#   2. Load template image by file name from the project
#   3. Match template in screenshot (scale sweep, then SIFT/FLANN)
#   4. Return center coordinates
press_element("button_template.png")
```

Optics finds a template by its file name anywhere in the project. An `input_templates/` folder is a common place for them. See [Project files](https://optics-framework.org/docs/reference/files).

**Image with index:**

```python
# Get second occurrence of template
press_element("icon_template.png", index=1)
```

**Image with AOI:**

```python
# Search for image only in top-right quarter
press_element(
    "button_template.png",
    aoi_x="50",      # Start at 50% from left
    aoi_y="0",       # Start at top
    aoi_width="50",  # 50% width
    aoi_height="50"  # 50% height
)
```

**Template organization:**

```text
project/
└── test_data/
    └── input_templates/
        ├── button_template.png
        ├── login_form.png
        └── submit_button.png
```

**Performance:**

- Slowest strategy, mostly in screenshot capture
- Template correlation runs on a reduced copy of the screen (tens of milliseconds)
- SIFT and FLANN run only when correlation finds nothing
- Performance improves with AOI

## Strategy selection

### Automatic selection

Optics selects strategies automatically from:

1. **Element type**: `utils.determine_element_type` reads it from the element identifier
   - XPath: `//button[@id='submit']`
   - Text: `"Submit Button"`
   - Image: `button_template.png`

2. **Element source capabilities**: Checked via `supports()` method
   - Does element source implement required methods?
   - Is the method actually implemented (not a stub)? `LocatorStrategy._is_method_implemented` reads the source of the method and treats a body that raises `NotImplementedError` as a stub.

3. **Priority ordering**: Strategies tried in priority order, inside each element source
   - Higher priority = tried first
   - Lower priority = fallback option

The element types and the strategies that take them:

| Element | Type | Strategies |
|---|---|---|
| Starts with `text_only:` or `text=` | Text | `TextElementStrategy` (not for `text_only:`), `TextDetectionStrategy` |
| Ends with `.jpg`, `.jpeg`, `.png` or `.bmp` | Image | `ImageDetectionStrategy` |
| Starts with `css=` | CSS | `TextElementStrategy` |
| Starts with `xpath=`, `/`, `//` or `(` | XPath | `XPathStrategy` |
| Starts with `id:` | ID | None of the four |
| Starts with `android.` or `xcui` | Class | `TextElementStrategy` |
| Has `[` and `]`, starts with `.`, is a `#id`, or starts with a common HTML tag followed by `[`, `#` or `.` | CSS | `TextElementStrategy` |
| Anything else | Text | `TextElementStrategy`, `TextDetectionStrategy` |

### Strategy factory

**Location:** `optics_framework/common/strategies.py`

The `StrategyFactory` makes strategies from what each element source can do.

```python
class StrategyFactory:
    def __init__(self, text_detection, image_detection, strategy_manager):
        self._registry = [
            (XPathStrategy, "XPath", {"strategy_manager": self.strategy_manager}, 1),
            (TextElementStrategy, "Text", {"strategy_manager": self.strategy_manager}, 2),
            (TextDetectionStrategy, "Text", {"text_detection": self.text_detection, "strategy_manager": self.strategy_manager}, 3),
            (ImageDetectionStrategy, "Image", {"image_detection": self.image_detection, "strategy_manager": self.strategy_manager}, 4),
        ]
```

The type in the registry is used only to decide whether to build the strategy for a source. At locate time, each strategy checks `supports()` again with the real element type. That is how `TextElementStrategy`, registered as "Text", also takes CSS and Class elements.

`PagesourceFactory` and `ScreenshotFactory` work the same way, with one entry each: `PagesourceStrategy` and `ScreenshotStrategy`.

### Strategy creation

```python
def create_strategies(self, element_source):
    strategies = [
        (cls(element_source, **args), priority)
        for cls, etype, args, priority in self._registry
        if cls.supports(etype, element_source)
    ]
    strategies.sort(key=lambda x: x[1])  # Sort by priority
    return [strategy for strategy, _ in strategies]
```

## Fallback mechanism

### Multi-level fallback

Optics falls back at more than one level:

`How the manager falls back from one strategy to the next`:

```mermaid
graph TB
    A[Action Request] --> B[StrategyManager]
    B --> C{Try Strategy 1}
    C -->|Success| D[Return Result]
    C -->|Fail| E{Try Strategy 2}
    E -->|Success| D
    E -->|Fail| F{Try Strategy 3}
    F -->|Success| D
    F -->|Fail| G{Try Strategy 4}
    G -->|Success| D
    G -->|Fail| H[All Strategies Failed]
```

### Strategy fallback flow

`StrategyManager.locate` is a lazy generator. It runs one strategy, yields its result, and runs the next strategy only when the caller asks for another result. A strategy that raises an error is skipped. If no strategy yields a result, `locate` raises `E0201`.

```python
def locate(self, element: str, aoi_x=None, aoi_y=None, aoi_width=None, aoi_height=None, index: int = 0) -> Generator[LocateResult, None, None]:
    self.require_screen()
    effective_element, text_only = utils.parse_text_only_prefix(element)
    element_type = utils.determine_element_type(element)
    internal_logger.info(f"Locating element: {element} of type: {element_type}...")
    use_aoi = self._validate_aoi(aoi_x, aoi_y, aoi_width, aoi_height)

    yielded = False
    last_error: Exception | None = None
    blind: Exception | None = None
    for strategy in self._locate_strategies(text_only):
        execution_logger.debug(f"Trying strategy: {type(strategy).__name__} for element: {effective_element}")
        try:
            locate_result = self._try_strategy_locate(
                strategy, effective_element, element_type, use_aoi, aoi_x, aoi_y, aoi_width, aoi_height, index
            )
        except Exception as e:
            raise_if_session_dead(e)
            if text_only or not _is_unconfigured_ocr(strategy):
                last_error = e
            blind = e if _is_blind(e) else blind
            continue
        if locate_result:
            yielded = True
            yield locate_result
    if not yielded:
        hint = self._ocr_not_tried_hint(element_type, text_only)
        raise blind or OpticsError(Code.E0201, message=f"Element '{element}' not found.{hint}", cause=last_error)
```

- `require_screen()` raises `E0501` when no `elements_sources` entry is enabled.
- A dead driver session stops the search at once (`raise_if_session_dead`).
- When OCR has no engine and a native lookup also ran, the OCR error does not hide the native miss. The message then ends with "OCR was not tried: no text_detection engine is configured."
- A camera that does not see the device screen (`E0305`) is raised in place of `E0201`.
- `_try_strategy_locate` records each attempt with `execution_tracer.log_attempt`.

### Element source fallback

When you configure more than one element source, Optics tries them in order:

```python
# If AppiumScreenshot fails, try SeleniumScreenshot
element_source_fallback.capture()
```

A direct call on an `InstanceFallback` tries each instance in config order and returns the first result that does not raise. `StrategyManager.capture_screenshot` does not use this call. It tries each `ScreenshotStrategy` in turn and returns the first frame that is not black. A black frame or a failed capture is `E0303`. The screenshot strategies are kept in a set, so do not rely on their order.

### Vision model fallback

When you configure more than one vision model, Optics tries them in order:

```python
# If EasyOCR fails, try GoogleVision
text_detection_fallback.detect_text(screenshot)
```

The next engine runs only when the one before it raises an error. An engine that returns no match ends the call.

### Assertion polling

`assert_presence` and `assert_visibility` poll in rounds until the timeout. Each round gives every strategy that can take the locator one non-waiting `check_elements()` call on the elements not yet found, cheap sources first, and stops at the first round that meets the rule (`all` counts each element once any source has found it). Rounds start at most once every 0.5 s, and the first round always runs in full. OCR and template matching share one screenshot taken in that round and run at most every 2 s while a cheaper strategy can still find every pending element.

- The rule must be `any` or `all`. Another value raises `E0205`.
- If no strategy can take any of the elements, or `all` is set and some element has no strategy that can take it, the assertion raises `E0201` at once.
- A strategy that raises an error leaves the poll for the rest of the assertion.
- If the round's screenshot is almost the same as the last one they read, OCR and template matching skip that round.
- For `assert_visibility`, `XPathStrategy` and `TextElementStrategy` take part only when the source implements `assert_elements_visible`.

### Screen capture caching

Inside one keyword, `one_capture_per_source()` captures each source's screen once and shares the frame between strategies. Around each action, `changing_screen()` drops the held frames, so every capture after the action is new. Each OCR engine also remembers its last 4 reads (`FRAMES_REMEMBERED`), so a second `detect_text` on the same pixels returns the earlier result.

## Self-healing action loop

**Location:** `optics_framework/api/action_keyword.py`

The `@with_self_healing` decorator is gone. Every keyword in `ActionKeyword` that locates an element now calls `_locate_and_act` with an `act` function. `_locate_and_act` locates the element through the strategies, gives each result to `act`, and goes to the next result when `act` fails. When nothing works, AI self-heal gets one chance before the keyword raises an error.

The methods:

| Method | What it does |
|---|---|
| `_locate_and_act(element, func_name, act, intent_args, index, aoi, allow_covered, value)` | Takes a screenshot, reads the AOI, saves screenshots, and starts `StrategyManager.locate`. |
| `_act_until_success(results, act, element, func_name, intent_args, screenshot_np, allow_covered, value)` | Runs `act` on each result until one succeeds. |
| `_act_on(result, act, element, func_name, allow_covered)` | Runs `act` on one result and returns its value or its error. |
| `_after_failed_acts(error, unsupported, element, func_name, heal)` | Picks the error, or lets AI self-heal finish the step. |

### Action flow

`How a keyword locates an element and falls back on failure`:

```mermaid
sequenceDiagram
    participant Action as Action Method
    participant Decorator as _locate_and_act
    participant SM as StrategyManager
    participant Strategy as Strategy
    participant Driver as Driver
    participant Utils as Utils

    Action->>Decorator: press_element(element, aoi_x, aoi_y, ...)
    Decorator->>SM: capture_screenshot()
    SM-->>Decorator: screenshot
    Decorator->>Utils: save_screenshot() or annotate_aoi_region()
    Utils-->>Decorator: saved
    Decorator->>SM: locate(element, aoi_x, aoi_y, ...)
    SM->>Strategy: try strategies
    Strategy-->>SM: LocateResult
    SM-->>Decorator: results (generator)
    loop For each result
        Decorator->>Driver: act(result.value)
        alt Success
            Driver-->>Decorator: success
            Decorator-->>Action: return result
        else Failure
            Driver-->>Decorator: exception
            Decorator->>Decorator: log error, try next
        end
    end
    alt All failed
        Decorator->>Decorator: AI self-heal, if enabled
        Decorator-->>Action: raise OpticsError
    end
```

### Implementation details

**Signature:**

```python
def _locate_and_act(
    self,
    element: Any,
    func_name: str,
    act: Callable[[Any], Any],
    intent_args: tuple = (),
    index: str = "0",
    aoi: Tuple[str, str, str, str] = ("0", "0", "100", "100"),
    allow_covered: bool = False,
    value: str | None = None,
) -> Any:
```

- `intent_args` are the keyword's action parameters. They feed the AI self-heal breadcrumbs.
- `value` is the text the keyword types. AI self-heal can choose where it goes, but never sees it.
- `allow_covered` lets a keyword that only reads an element use a web point that another element covers.

**Step-by-step process:**

**1. Screenshot capture:**

```python
self.strategy_manager.require_screen()
with one_capture_per_source():
    screenshot_np = self._capture_screenshot_safe()
```

If the capture fails, the keyword continues without a screenshot.

**2. AOI parameter extraction:**

```python
aoi_x = _parse_aoi_param(aoi[0], 0)
aoi_y = _parse_aoi_param(aoi[1], 0)
aoi_width = _parse_aoi_param(aoi[2], 100)
aoi_height = _parse_aoi_param(aoi[3], 100)
is_aoi_used = not (aoi_x == 0 and aoi_y == 0 and aoi_width == 100 and aoi_height == 100)
```

`_parse_aoi_param` returns the default for `None`, an empty string, `'None'` or `'none'`, and a `float` for any other value.

**3. Screenshot saving:**

```python
if is_aoi_used and screenshot_np is not None and self.capture_output_dir is not None:
    _maybe_save_aoi_screenshot(
        screenshot_np, aoi_x, aoi_y, aoi_width, aoi_height,
        self.capture_output_dir, func_name,
    )
self._save_screenshot_if_available(screenshot_np, f"pre_{func_name}")
```

`_maybe_save_aoi_screenshot` calls `utils.annotate_aoi_region` and saves the result as `f"{func_name}_with_aoi"`. The screenshot before the action is always saved as `pre_<func_name>` when there is one.

**4. Element location:**

```python
if is_aoi_used:
    return strategy_manager.locate(element, aoi_x, aoi_y, aoi_width, aoi_height, index=index)
return strategy_manager.locate(element, index=index)
```

**5. Result iteration** (`_act_until_success`):

```python
result_count = 0
results_iter = iter(results)
while True:
    try:
        result = next(results_iter)
    except StopIteration:
        break
    except OpticsError as e:
        locate_error = e
        break
    result_count += 1
    _save_annotated_for_result(result, screenshot_np, self.capture_output_dir, func_name)
    ret, error = self._act_on(result, act, element, func_name, allow_covered)
    if error is not None:
        last_exception = error
        unsupported = unsupported and _unsupported(error)
        continue
    keyword_outcome.acted(result.strategy)
    self._record_successful_step(func_name, element, intent_args, value)
    return ret
```

- The results are lazy, so a later strategy locates only after `act` fails on the one before.
- `_act_on` runs `act` with the driver that belongs to the result's element source (`self.driver.routed_to(...)`), inside `changing_screen()`.
- `_act_on` refuses a web point that another element covers, unless `allow_covered` is set. The next result then gets a try.
- `E0405` (the action ran, but the screen does not show its result) and `E0106` (the session is gone) stop the loop at once. Another result cannot do better.
- Any other error is logged as `Action '<func_name>' failed with <Strategy>: <error>`, and the next result gets a try.

**6. Error handling:**

```python
if locate_error is not None and locate_error.code == Code.E0305:
    raise locate_error
if result_count == 0:
    if self._ai_self_heal(element, func_name, intent_args, screenshot_np, value):
        return None
    raise OpticsError(
        Code.E0201,
        message=f"No valid strategies found for '{element}' in '{func_name}'",
        cause=locate_error,
    ) from locate_error
```

`_after_failed_acts` handles the case where results came back but every `act` failed:

```python
if error is None:
    raise OpticsError(Code.E0801, message=f"Unexpected failure: No results or exceptions for '{element}' in '{func_name}'")
if unsupported:
    raise error
if heal():
    return None
raise OpticsError(
    Code.X0201,
    message=f"All strategies failed for '{element}' in '{func_name}': {error}",
    cause=error,
)
```

When every driver refused the action itself (`E0105`, keyword not supported on this platform), Optics raises that error. No other element and no self-heal can do it.

### Key features

**1. Automatic screenshot capture:**

- Captures screenshot before element location
- Saves screenshot for debugging
- Annotates AOI region if used

**2. Strategy-based location:**

- Uses StrategyManager to locate element
- Supports multiple location strategies
- Handles AOI parameters automatically

**3. Multiple result handling:**

- Iterates through all successful strategy results
- Tries each result until one succeeds
- Continues on failure (self-healing)

**4. Error aggregation:**

- Logs each failure with strategy information
- Preserves last exception for error message
- Provides detailed error context

**5. Screenshot saving:**

- Saves screenshots to execution output directory
- Annotates AOI regions for debugging
- Uses function name for screenshot filename
- Saves the match: the annotated frame from OCR or image matching, or the element box from a native source

**6. AOI integration:**

- Reads AOI parameters from the `aoi` tuple
- Passes AOI to strategy manager
- Annotates screenshots with AOI region

**7. AI self-heal:**

- Runs only when no strategy finds the element or every action fails
- Off by default. See [AI self-heal](https://optics-framework.org/docs/elements/self-heal).

### Usage example

**Basic usage:**

```python
def press_element(
    self, element: str, repeat: str = "1", offset_x: str = "0", offset_y: str = "0", index: str = "0", aoi_x: str = "0",
    aoi_y: str = "0", aoi_width: str = "100", aoi_height: str = "100", event_name: Optional[str] = None
    ) -> None:
    count, dx, dy = _repeat(repeat), _offset(offset_x, "offset_x"), _offset(offset_y, "offset_y")

    def act(located: Any) -> None:
        if isinstance(located, tuple):
            x, y = located
            self.driver.press_coordinates(x + dx, y + dy, event_name=event_name)
        else:
            self.driver.press_element(located, count, event_name=event_name)

    self._locate_and_act(
        element, "press_element", act, (repeat, offset_x, offset_y),
        index=index, aoi=(aoi_x, aoi_y, aoi_width, aoi_height),
    )
```

**With AOI:**

```python
# Press element in specific region
press_element(
    "submit_button",
    aoi_x="50",      # Right half of screen
    aoi_y="0",       # Top
    aoi_width="50",  # 50% width
    aoi_height="100" # Full height
)
```

### Error scenarios

**1. No strategies available:**

```python
# Error: E0201 - No valid strategies found
# Occurs when no strategies can locate the element
```

**2. All strategies failed:**

```python
# Error: X0201 - All strategies failed
# Occurs when all strategies tried but all failed
# Includes last exception as cause
```

**3. Unexpected failure:**

```python
# Error: E0801 - Unexpected failure
# Occurs in unexpected scenarios
```

**4. Other errors that end the step:**

- `E0305`: the camera does not see the device screen
- `E0105`: every driver refused the action
- `E0405` or `E0106`: the action ran without the expected result, or the session is gone
- `E0205`: invalid AOI parameters

See [Error codes](https://optics-framework.org/docs/reference/errors).

### Debugging support

**Screenshot files:**

- Saved to `execution_output_path` (the session's capture directory)
- Named `<timestamp>-<name>.jpg`, where the name comes from the keyword's function name:
  - `pre_press_element`: the screen before the action
  - `press_element_with_aoi`: the screen with the AOI annotated
  - `press_element_image_detection_result`: the frame annotated by OCR or image matching
  - `press_element_element_detection_result`: the element box from a native source

**Logging:**

- Each strategy attempt is logged
- Failures include strategy name and error
- Execution logger shows location attempts

**Error messages:**

- Include element name
- Include function name
- Include strategy that failed
- Include original exception

### Best practices

1. **Always use `_locate_and_act`**: Use it in every keyword that acts on an element
2. **Handle the located value**: Check if tuple (coordinates) or object (element)
3. **Provide context**: Include meaningful error messages
4. **Save screenshots**: Enable screenshot saving for debugging
5. **Use AOI when appropriate**: Improve performance and accuracy

### Integration points

- **StrategyManager**: For element location
- **Utils**: For screenshot saving and AOI annotation
- **Driver**: For actual action execution
- **Error System**: For error reporting
- **Logging**: For debugging information
- **AI self-heal**: For the last attempt when every strategy fails

## Screenshot strategies

### ScreenshotStrategy

Captures screenshots from element sources.

#### Features

- Black screen detection (`utils.is_black_screen`)
- Screenshot validation: a black or failed capture raises `E0303`
- Multiple element source fallback

A source gets a `ScreenshotStrategy` when it implements `capture()`.

## Page source strategies

### PagesourceStrategy

Gets the page source from element sources.

#### Features

- Page source extraction: `capture_pagesource()` returns `(page_source, timestamp)`, or raises `E0403`
- Interactive element detection: `get_interactive_elements(filter_config, compact)`
- Multiple format support (XML, HTML)
- Element hierarchy navigation

A source gets a `PagesourceStrategy` when it implements `get_page_source()`. `compact` goes only to sources whose `get_interactive_elements` accepts it, so an older source with the signature `get_interactive_elements(filter_config)` still works.

`StrategyManager.get_interactive_elements` uses the first source that lists elements, then merges in what each supplementing source adds. When no source can list elements, it uses the OCR rows that `TextDetectionStrategy.screen_elements()` reads. OCR guesses below a confidence of 0.2 are left out. When it gets no list at all, it raises `E0202`.

## Strategy result handling

### LocateResult

Holds a location result and the strategy that found it:

```python
class LocateResult:
    def __init__(
        self,
        value: Union[object, Tuple[int, int]],
        strategy: LocatorStrategy,
        annotated_frame: Optional[Any] = None,
    ):
        self.value = value  # Coordinates or element object
        self.strategy = strategy  # Which strategy found it
        self.annotated_frame = annotated_frame
        self.is_coordinates = isinstance(value, tuple)
```

`annotated_frame` is the screenshot with the match boxed. Vision strategies return it in a `LocateValueWithFrame(value, annotated_frame)`.

### Result types

1. **Coordinates**: `(x, y)` tuple for direct coordinate-based actions
2. **Element object**: Element object for driver-specific actions

## Best practices

### Strategy selection

1. **Use XPath when possible**: Fastest and most reliable
2. **Fallback to vision**: When UI structure is unknown
3. **Configure multiple models**: For maximum resilience
4. **Set appropriate timeouts**: For element appearance

### Performance optimization

1. **Priority ordering**: Fastest strategies first
2. **Caching**: Each source's screenshot is captured once per keyword, and each OCR engine remembers its last reads
3. **Early exit**: Stop trying when first strategy succeeds
4. **One at a time**: Strategies run in sequence, never in parallel. A later strategy runs only when the one before it fails.

### Error handling

1. **Graceful degradation**: Try next strategy on failure
2. **Error context**: Keep the last error as the cause of the final error, and log each failure
3. **Screenshot capture**: Save the screen before each action, and the match that each strategy found
4. **Detailed logging**: Log strategy attempts and results

## Extension guide

### Adding a new strategy

1. Create class extending `LocatorStrategy`:

```python
class CustomStrategy(LocatorStrategy):
    def __init__(self, element_source: ElementSourceInterface, strategy_manager=None):
        self._element_source = element_source
        self._strategy_manager = strategy_manager

    @property
    def element_source(self) -> ElementSourceInterface:
        return self._element_source

    def locate(self, element: str, index: int = 0):
        # Implementation
        pass

    def check_elements(self, elements: list[str], visible: bool, frame):
        # One look per element, no waiting
        return [False] * len(elements), None

    @staticmethod
    def supports(element_type: str, element_source: ElementSourceInterface) -> bool:
        return element_type == "Custom" and ...
```

2. Register in `StrategyFactory`:

```python
self._registry = [
    ...,
    (CustomStrategy, "Custom", {"strategy_manager": self.strategy_manager}, 5),  # Priority 5
]
```

3. Make `utils.determine_element_type` return your type for your elements.

4. Strategy will be automatically used when:
   - Element type matches
   - Element source supports it
   - Higher priority strategies fail

For a strategy that can search inside an AOI, add `locate_with_aoi(element, aoi_x, aoi_y, aoi_width, aoi_height, index=0)`. Without it, `StrategyManager` filters your results by their centre point.

## Debugging strategies

### Enable debug logging

Set `log_level: DEBUG` in config.yaml. See [Logging](https://optics-framework.org/docs/reference/config#logging). The level applies to both loggers, `optics.internal` and `optics.execution`, and to their console handlers. A `setLevel` call on one logger alone does not show debug lines, because the console handler keeps its own level.

### Strategy execution log

The framework logs:

- Which strategies are tried
- Which strategy succeeded
- Why strategies failed
- Screenshot locations

### Visual debugging

Screenshots are saved with the keyword name, not the strategy name. See [Debugging support](#debugging-support) for the names.

## Related documentation

- [Components](https://optics-framework.org/docs/contribute/components): component architecture and the Optics class
- [Engines](https://optics-framework.org/docs/contribute/engines): engine implementations
- [Execution](https://optics-framework.org/docs/contribute/execution): execution flow and test context
- [Error handling](https://optics-framework.org/docs/contribute/errors): error codes and handling
- [Logging](https://optics-framework.org/docs/contribute/logging): logging system
- [Architecture decisions](https://optics-framework.org/docs/contribute/decisions): Strategy pattern rationale
- [Extending](https://optics-framework.org/docs/contribute/extending): creating custom strategies
- [Self-healing action loop](#self-healing-decorator): detailed implementation
- [AOI support](#aoi-support): Area of Interest system
