Manual RTL testing vs automated RTL testing

An automated RTL suite can pass while an Arabic customer still sees the wrong illustration, an unnatural rejection message, or a settlement action whose arrow implies the opposite movement. A manual review can approve the same screen while missing a one-pixel document overflow, a route that loses dir="rtl" after sign-in, or a fallback state that nobody opened.

Neither result is contradictory. The two methods answer different questions. Automated RTL testing compares observable output with an explicit rule. Manual RTL testing asks whether the output makes sense to a person, in context, across interactions that may not have a stable machine-readable answer.

The useful choice is not manual or automated. It is deciding which claims a test can prove, which decisions require judgment, and how evidence moves between the two.

The boundary is evidence versus judgment#

Automation needs an oracle: a condition that is unambiguously true or false. An Arabic route either has dir="rtl" or it does not. Its root element either overflows the viewport or it does not. A field has a computed direction, a button has an accessible name, and a formatter returns a value that can be compared with an approved fixture.

Human review is needed when the acceptance rule depends on meaning or intent. A left-pointing arrow might mean “next” in a right-to-left sequence, “back” in browser history, or “move this claim to the left column.” Geometry cannot decide which interpretation the product intends. A person must understand the control before judging its direction.

That split produces a more useful comparison than a list of advantages and disadvantages:

What automated RTL testing catches well#

Automate a finding when the product has a stable rule, the relevant state can be created reliably, and the failure can be observed without interpretation. RTL products contain many such rules.

Document metadata is a clear example. Every Arabic route can assert an Arabic lang value and an RTL dir value on the root element. R001 reports document direction that is missing, contradicted or declared only in CSS. R002 reports a missing, non-Arabic or contradictory html[lang]. Those findings are separate because language and direction are separate inputs to the browser and assistive technology.

Rendered layout also exposes measurable facts:

• direction-sensitive physical CSS can be reviewed or linted against the component's RTL implementation;
• the document's scrollWidth can be compared with its clientWidth at fixed viewports;
• text-bearing boxes can be checked for overlap or clipping;
• computed font families can be compared after webfonts finish loading;
• Arabic text inputs and inherently LTR values can be checked for the direction required by the field contract;
• known routes can be searched for untranslated controls and attributes;
• expected digit, date and currency output can be compared with market fixtures.

These are not merely visual preferences. R030 reports direction-sensitive physical properties without an RTL override. R040 reports document-level horizontal overflow. R061 reports Arabic-Indic and Western digits mixed in the same context. Each has a defined condition that a rendered-page scan can report consistently.

The limit is reachability. A scanner, browser test or visual suite can inspect only the states it opens. If the Arabic error appears after a particular combination of policy type, evidence format and server response, the test must create that combination. “Covered by automation” is not a property of a route. It is a property of a named state, data fixture, viewport and assertion.

What automation can only approximate#

Some signals look objective but are only proxies.

A high volume of Latin text on an Arabic route can indicate untranslated content. It can also be valid content: an email address, a vehicle identification number, a company name or a model code. R010 reports visible Latin-script body text above its thresholds. The finding tells a reviewer where to look; it does not translate the text or decide whether the Latin run is intentional.

Geometry has the same limitation. An automated check can report that two text-bearing elements overlap. It cannot always infer whether the elements are duplicates during a transition, labels deliberately layered in a chart, or controls that have become unreadable. The machine provides a reproducible location and measurement. A person classifies the product impact.

Visual regression can detect more shapes than a narrow DOM assertion, but it inherits the approved image's mistakes. If the initial baseline shows an unmirrored progress illustration, every unchanged run passes. If a timestamp, font-rendering difference or animated placeholder changes pixels, the suite can fail even though the RTL behavior did not regress.

Language checks are also approximate unless the expected copy is controlled. A test can verify that the claim-status button equals an approved Arabic string. It cannot determine that the approved string sounds legalistic, omits a crucial distinction, or uses terminology inconsistent with the policy document. String presence is deterministic. Language quality is not.

What still needs manual RTL testing#

Manual testing earns its place where the tester must understand purpose, read the language, or interact with behavior that is difficult to model faithfully.

Meaning determines what should mirror#

Direction is semantic before it is geometric. A chevron that advances through an RTL sequence generally points toward the next item in that sequence. A media play icon does not reverse. A diagram of a physical process might have a real-world direction that should stay fixed. The guide to what should be mirrored in RTL covers those categories in depth.

Automation can assert the approved variant once the team has made the decision. It cannot originate the decision from the icon's coordinates. R025 reports a directional icon pointing against the sequence its control advances, which is useful when the sequence is clear. Ambiguous controls still need a reviewer to define their meaning.

Translation quality is not string coverage#

A complete catalog prevents missing keys. It does not establish that an Arabic customer understands the message or knows what to do next. Reviewers need the rendered state, surrounding actions and source intent.

Consider an insurance evidence upload. The Arabic message for an unsupported file may be grammatically correct but fail to name the accepted file types. Another translation may preserve every source word while putting the recovery instruction after a secondary legal note. A fluent reviewer should submit the invalid file, read the message in place, identify the next action and confirm that terminology matches the rest of the claims journey.

R071 reports untranslated form labels, validation or help text. It does not judge the quality of translated validation, focus recovery, or whether the advice solves the user's problem.

Editing behavior needs hands-on interaction#

Mixed-direction editing involves the caret, selections, deletion boundaries, pasted content and input-method behavior. DOM text and screenshots capture only snapshots. They do not show whether a reviewer can edit the middle group of a policy reference without losing their place, or whether pasting a vehicle plate changes the field direction.

Test real input with the keyboard. Move the caret through the value, select part of it, delete around punctuation, paste from an external document and submit. Repeat after validation has changed the DOM. For mixed Arabic and machine-readable values, this manual pass finds interaction costs that a final-value assertion cannot express.

Assistive technology needs a human listener#

Automated accessibility checks can confirm attributes, names, roles, language tags and focus targets. They cannot decide that a spoken sequence is understandable, that a mixed-language announcement switches voice appropriately, or that a status update provides enough context.

The DOM's logical order matters more than the visual position here. A layout can look correct while focus moves through controls in an order that fights the task. R081 reports a flex row-reverse implementation that fakes RTL instead of using document direction, where tab order fights visual order. For complex widgets, a keyboard and screen-reader pass still has to confirm the experience rather than infer it from a screenshot.

Some failures need both methods#

The strongest workflow assigns different parts of one risk to automation and people. It does not label an entire feature “manual” or “automated.”

Take a claim reference written as 7364-918-27. A test can assert that its component uses an isolated value boundary, preserve a visual baseline and check that copying yields the authored string. A reviewer must still know whether the value is an immutable identifier or a numeric range. The same displayed reordering can be correct reading order for a range and corrupt for an identifier. The bidirectional text guide explains that distinction and the relevant markup. R021 reports unisolated bidi hazards such as phones, emails and Latin tokens inside Arabic text.

Other shared risks divide the same way:

Start with a release risk map#

Do not split effort by percentage. Split it by the changed behavior and the strength of its test oracle.

Imagine a release for an insurance claims portal. It changes the evidence uploader, the coverage summary card, the settlement formatter and the status timeline. Map each changed surface before choosing tests.

This map prevents two wasteful outcomes. One is sending a fluent reviewer a build whose missing direction and overflow could have failed earlier. The other is accepting a green pipeline as proof that the Arabic journey makes sense.

Give automation explicit oracles#

An automated test should state the contract it is proving. “Check the Arabic page” is not a test case.

For root metadata, define the routes and assert both attributes:

// page and expect are supplied by the browser test runner.
const arabicClaimRoutes = [
  "/ar/claims/new",
  "/ar/claims/evidence",
  "/ar/claims/settlement",
];

for (const route of arabicClaimRoutes) {
  await page.goto(route);
  const root = page.locator("html");

  await expect(root).toHaveAttribute("lang", /^ar(?:-|$)/);
  await expect(root).toHaveAttribute("dir", "rtl");
}

This proves live root state on those three URLs. It does not prove that sign-in redirects, language switching or a client-side error boundary preserve the attributes. Add those transitions as separate cases if the release can affect them.

For document overflow, wait for the exact state under test. Fonts, evidence thumbnails and validation messages can all change geometry after initial navigation.

// page and expect are supplied by the browser test runner.
// openRejectedEvidenceState creates the named API response and waits for rendering.
await openRejectedEvidenceState(page);
await page.evaluate(() => document.fonts.ready);

const overflowPixels = await page.evaluate(() => {
  const root = document.documentElement;
  return Math.max(0, root.scrollWidth - root.clientWidth);
});

expect(overflowPixels).toBeLessThanOrEqual(1);

The one-pixel allowance is a test-suite decision, not a web standard. Document it and keep it stable. If the product deliberately has a horizontally scrolling evidence strip, measure the document root separately from that component. A passing root check does not establish that the inner strip is keyboard accessible or understandable.

Formatter tests should assert product decisions rather than whatever the runtime happens to choose. Pin the locale extensions and options in the formatter, then inspect them in the test.

// expect is supplied by the test runner.
const settlementFormat = new Intl.NumberFormat("ar-AE-u-nu-latn", {
  style: "currency",
  currency: "AED",
  minimumFractionDigits: 2,
  maximumFractionDigits: 2,
});

const options = settlementFormat.resolvedOptions();
expect(options.numberingSystem).toBe("latn");
expect(options.currency).toBe("AED");
expect(options.minimumFractionDigits).toBe(2);
expect(options.maximumFractionDigits).toBe(2);

This test proves configuration, not visual order or agreement with policy. Add an approved rendered fixture in the environments the product supports, and have the market owner approve the rule itself. Runtime locale data is not a substitute for a product requirement.

Build manual tests as charters#

“Review the Arabic page” is just as weak as “check it automatically.” A manual charter gives the tester a purpose, controlled data, important states and decisions to make while leaving room to explore.

A useful charter for the insurance evidence flow could say:

Purpose: judge whether an Arabic claimant can understand and recover from evidence upload failures.
Data: a valid image, an oversized document, an unsupported archive and a filename containing Arabic plus the code POL-Z7-482.
States: initial, uploading, rejected by the client, rejected by the server, retrying and complete.
Review: wording, reading order, action hierarchy, focus recovery, progress meaning, keyboard use and narrow-screen wrapping.
Evidence: route, viewport, fixture, screenshot or recording, source text, observed Arabic text and expected behavior.

The charter does not tell the reviewer to click every control mechanically. It directs attention to the release risk and records enough context to reproduce a failure.

Use specialist review only where it adds information. A fluent Arabic reviewer should judge terminology, tone and comprehension. A frontend engineer should inspect computed styles and component behavior. An accessibility tester should verify keyboard and assistive-technology interaction. One person can cover more than one role, but the sign-off should name the judgment being made.

For teams without an Arabic speaker in engineering, the guide to QA Arabic without speaking Arabic shows how to separate structural checks from language approval without pretending that one replaces the other.

Reach the states that matter#

Coverage is often lost before the assertion runs. Happy-path seed data opens a loaded screen, so tests never render the long permission error, expired settlement offer, empty timeline or partial-coverage notice where RTL defects live.

Create states deliberately. API fixtures, request interception, component stories or seeded accounts can make rare responses repeatable. Give each fixture a product meaning, not just a status code. claim-partially-covered-with-long-note is more useful than error-3 because reviewers know what the state is intended to communicate.

Automation should establish that the state remains reachable. Manual review should then use the same fixture. That shared setup removes a damaging ambiguity: the reviewer and pipeline are looking at the same content and component state.

Do not claim coverage from a screenshot inventory alone. A screen can have captures for desktop and mobile while its open menu, validation failure and delayed font state remain untested. Track coverage as a tuple:

route + state + fixture + viewport + interaction + assertion

If one part is absent, write down the gap. A known manual gap is easier to prioritize than a vague claim that the page has “RTL coverage.”

Use screenshots for geometry, not approval#

Visual regression is useful when the baseline, data and rendering conditions are controlled. Capture focused components for spacing and wrapping, plus a small number of journey-level screens where relationships matter.

Stabilize fonts, animation, time, user data and network responses before capture. Compare the same state in LTR and RTL, but do not expect a mathematical mirror. Text lengths differ, non-directional icons stay fixed, and some data remains LTR. The approved result is a semantic RTL counterpart, not a horizontally flipped image.

When a screenshot changes, review the relationship that moved:

• Did logical start and end placement change as intended?
• Did Arabic expansion push a control over its label?
• Is the webfont present, or is fallback changing line breaks?
• Does a directional icon still match the sequence?
• Is content clipped, hidden or merely wrapped differently?
• Does the same change appear in LTR, suggesting a shared component regression?

Run deterministic gates before human review#

Manual review should start from a build that has passed the cheap, repeatable checks. Otherwise a specialist spends time reporting missing translations, broken root direction and obvious overflow instead of evaluating meaning.

A practical sequence is:

1. At pull request time, run catalog validation, direction-sensitive CSS review, component assertions and targeted browser tests from the changed surfaces.
2. On the release candidate, run the representative Arabic route set with controlled states, fixed viewports, geometry checks and visual comparisons.
3. Give the resulting candidate and fixture list to language, accessibility and product reviewers for their charters.
4. After deployment, run a small smoke pass against the real environment to catch configuration, asset and routing differences.

This is not a waterfall. A manual finding should improve an automated assertion when the team can define a stable oracle. If a reviewer finds that a server rejection loses dir="rtl", create the fixture and regression test. If the reviewer finds that the approved phrase is ambiguous, keep language review in the loop even after updating the text.

Triage disagreements instead of voting#

A green automated run and a failed manual review do not cancel each other out. They describe different facts.

When automation passes and a person finds a defect, ask:

• Did the suite reach the same state and fixture?
• Was the acceptance rule measurable but missing?
• Did the baseline already contain the defect?
• Is the issue semantic, linguistic or interaction-based and therefore still a manual judgment?

When automation fails and the reviewer sees no problem, inspect the oracle:

• Is the detected Latin content an intentional identifier?
• Is the overflow inside an intended scroll region rather than the document?
• Did font loading, animation or live data make the screenshot nondeterministic?
• Has the product contract changed without updating the fixture?

Do not resolve the disagreement by marking the automated result “false positive” with no follow-up. Either refine the assertion, document an intentional exception close to the rule, or fix the product. Repeated unexplained exceptions teach the team to ignore the suite.

Automate the next repeatable decision#

Manual findings are a source of future test design. Automate a finding when it recurs in a component family, can be reproduced with controlled data, and has an acceptance rule that does not require a person to reinterpret it each run.

Good candidates include a modal that repeatedly loses root direction through a portal, a card whose long Arabic notice clips at a known breakpoint, or a formatter that falls back to a different numbering system. These can become component assertions, fixtures or rendered-page gates.

Keep manual ownership when the decision changes with context. Illustration direction, legal tone, terminology choices and information hierarchy do not become objective because someone captured a baseline. Automation can confirm that the approved asset or copy version shipped, while approval itself remains human.

Review stale tests as the product changes. A test can keep passing after the route it protects stops serving real users, or after a new component replaces the old one. The right metric is not the number of cases. It is whether release risks have reachable states, trustworthy oracles and named human decisions.

Assign ownership to the evidence#

The split works only when failed evidence reaches someone who can act on it.

• Frontend engineers own document direction, logical layout, component behavior, field direction and the regression tests attached to fixes.
• QA engineers own state coverage, fixtures, reproducibility, exploratory charters and the trace between a release change and its evidence.
• Arabic reviewers own language accuracy, tone, terminology and comprehension in rendered context.
• Designers and product owners decide semantic mirroring, information hierarchy and market-specific product rules.
• Accessibility reviewers own keyboard and assistive-technology judgment where automated attributes are insufficient.

Record the result in terms of claims: what automation proved, what a person reviewed, which states remain out of scope and who accepted those gaps. That record makes the next release easier to scope and stops a generic green badge from standing in for actual Arabic QA.

Use the two methods as one feedback loop#

Manual RTL testing finds problems whose correctness depends on meaning, context and interaction. Automated RTL testing guards the rules that can be expressed and repeated. A mature workflow moves findings in both directions: automation prepares a clean candidate for human judgment, and human discoveries become regression tests whenever their expected outcome is stable.

A Ritla scan contributes the rendered-page findings that machines can report consistently, including missing direction, direction-sensitive CSS and document overflow. Use those results to clear deterministic defects before the manual charters begin, then use the Arabic product QA guide to place language, market and workflow decisions around that evidence.