The Selector Ladder: Why Your Browser Automation Breaks Every Week

“My click isn't working” is never the real problem. The real problem is: wrong selector, wrong timing, or wrong assumption about state. After months of running browser automation agents against production websites, we learned this the hard way.

LinkedIn regenerates its CSS classes every week. X's React app batches state updates asynchronously. A selector that worked yesterday will fail tomorrow. Not because the page changed semantically — but because the surface representation shifted underneath you.

Here's the reliability hierarchy we developed. We call it the Selector Ladder.

The Selector Ladder

Always try selectors in this order. Stop at the first one that works:

1. Playwright snapshot ref          [ref=e42]
2. ARIA role + accessible name      role=button[name="Submit"]
3. aria-label attribute             [aria-label="Close dialog"]
4. data-testid attribute            [data-testid="tweet"]
5. Semantic HTML                    button, input[type="submit"]
6. Text content                     text="Sign in"
7. CSS class (obfuscated = fragile) .x1abc123
8. Coordinate click (last resort)   (x: 450, y: 320)

Why this order matters:

• Snapshot refs — Most reliable, direct handle. But stale after re-render.
• ARIA attributes — Stable by design. Changing them breaks screen readers, so companies don't change them. LinkedIn's CSS classes are regenerated weekly. Their ARIA attributes haven't changed in years.
• data-testid — Maintained by frontend teams for their own Cypress/Playwright tests. X (Twitter) uses these extensively and they're remarkably stable.
• CSS classes — Fragile. Obfuscated on LinkedIn, minified on most React apps. If you're selecting by CSS class on a major platform, you're building a brittle system.
• Coordinates — Last resort. Viewport changes, A/B tests, and responsive breakpoints all break this. But sometimes it's the only option for shadow DOM or cross-origin iframes.

ARIA Selectors: The Gold Standard

Playwright's ARIA selectors are the most underused tool in browser automation:

// By role
page.getByRole('button', { name: 'Submit' })
page.getByRole('textbox', { name: 'Search' })
page.getByRole('link', { name: 'Sign in' })
page.getByRole('dialog')
page.getByRole('heading', { level: 1 })

// By label (finds input associated with <label>)
page.getByLabel('Email address')

// By placeholder
page.getByPlaceholder('Search tweets...')

// By text content
page.getByText('Sign in', { exact: true })
page.getByText(/sign in/i)  // regex, case-insensitive

Consider this HTML:

<!-- This HTML might change -->
<button class="x1abc123 xdef456">Submit</button>

<!-- But the ARIA semantics stay the same -->
<!-- Role: button. Name: "Submit" (from text content) -->
<!-- page.getByRole('button', { name: 'Submit' }) ALWAYS WORKS -->

The Five Waiting Strategies

Timing is as important as selection. Here are the five waiting strategies, in order of reliability:

1. waitForResponse (Network Signal)

The most reliable. Wait for the API call that fetches your data:

const [response] = await Promise.all([
  page.waitForResponse(r =>
    r.url().includes('/CreateTweet') && r.status() === 200
  ),
  page.click('[data-testid="tweetButtonInline"]')
])
const result = await response.json()
// You know EXACTLY when it's done — and get data back

2. waitForSelector (DOM Signal)

Wait for an element to appear or disappear:

await page.waitForSelector('[data-testid="tweet"]', {
  state: 'visible',
  timeout: 10000
})

3. waitForFunction (Custom Condition)

When no single element signals completion:

await page.waitForFunction(() => {
  const tweets = document.querySelectorAll('[data-testid="tweet"]')
  return tweets.length > 5
}, { timeout: 15000 })

4. waitForLoadState (Page Lifecycle)

For initial page loads. Avoid networkidle on pages with polling:

await page.goto('https://x.com/home')
await page.waitForLoadState('domcontentloaded')

5. setTimeout (Last Resort)

Only when you can't wait for a signal. We use 3s minimum for X, 4s for LinkedIn:

// Saves 1 round-trip vs. separate sleep call
await page.evaluate(() => new Promise(r => setTimeout(r, 3000)))

Self-Healing Selectors

Even with the Selector Ladder, things break. Our agents run a self-healing protocol when a selector fails:

async function clickWithHealing(page, selector, description) {
  try {
    await page.click(selector, { timeout: 5000 })
    return { healed: false }
  } catch (e) {
    // Take accessibility snapshot
    const snapshot = await page.accessibility.snapshot()

    // Find candidates by text similarity
    const candidates = await page.evaluate((desc) => {
      return Array.from(document.querySelectorAll(
        'button, a, input, [role="button"], [tabindex]'
      ))
      .map(el => ({
        text: el.textContent?.trim().slice(0, 100),
        aria: el.getAttribute('aria-label'),
        testid: el.getAttribute('data-testid'),
      }))
      .filter(el =>
        el.text?.toLowerCase().includes(desc.toLowerCase()) ||
        el.aria?.toLowerCase().includes(desc.toLowerCase())
      )
    }, description)

    // Use best candidate
    const best = candidates[0]
    const newSelector = best.testid
      ? `[data-testid="${best.testid}"]`
      : best.aria
        ? `[aria-label="${best.aria}"]`
        : `text="${best.text}"`

    await page.click(newSelector)
    console.log(`SELECTOR_HEALED: ${selector} → ${newSelector}`)
    return { healed: true, old: selector, new: newSelector }
  }
}

The key insight: when a selector fails, don't retry the same selector. Take a fresh accessibility snapshot, find the element by its semantic meaning (role + text), and persist the healed selector for next time.

We maintain a selector cache that maps broken selectors to their healed replacements. Over time, the system becomes more resilient — it learns which selectors are fragile and automatically substitutes stable ones.

The React/SPA Timing Trap

React creates a specific timing challenge because state updates are batched and asynchronous:

click() fires
  → onClick handler runs
  → setState() called
  → React schedules re-render (microtask queue)
  → Virtual DOM diffed
  → Real DOM updated
  → a11y tree updated (one more frame)

This means you can't read the DOM immediately after a click — the update hasn't propagated yet. For React-controlled inputs, there's an additional trap:

// WRONG — React might not see this
await page.fill('#email', 'user@example.com')

// RIGHT — triggers onChange per character
await page.type('#email', 'user@example.com', { delay: 50 })

// OR — use CDP Input.insertText (simulates OS-level input)
// This passes through React, Vue, Angular — all frameworks

We use Input.insertText via CDP for our X skill for exactly this reason — it simulates actual keyboard input through the OS event layer, which every framework handles correctly.

LinkedIn's Virtualized DOM

LinkedIn uses virtual scrolling — only ~10 posts are in the DOM at any time. Scroll down and older posts are removed from the DOM entirely.

async function extractAllPosts(page, targetCount = 20) {
  const posts = new Map()  // url → data (dedup by URL)

  while (posts.size < targetCount) {
    // Extract currently visible posts
    const visible = await page.evaluate(() =>
      Array.from(document.querySelectorAll('.feed-shared-update-v2'))
        .map(el => ({
          url: el.querySelector('time')?.closest('a')?.href,
          text: el.querySelector('.feed-shared-text')?.textContent?.trim(),
        }))
        .filter(p => p.url)
    )

    for (const post of visible) posts.set(post.url, post)
    if (posts.size >= targetCount) break

    // Scroll down — extract BEFORE scrolling or you lose the data
    await page.evaluate(() => window.scrollBy(0, window.innerHeight * 2))
    await page.evaluate(() => new Promise(r => setTimeout(r, 2000)))
  }
  return Array.from(posts.values())
}

The critical insight: always extract before scrolling. If you scroll without extracting, that data is gone. The DOM doesn't keep it around.

Patterns We Never Use

// ❌ CSS class on LinkedIn (obfuscated weekly)
await page.click('.artdeco-button--2')

// ❌ Index-based selection (breaks when order changes)
await page.click('button:nth-child(3)')

// ❌ XPath with position
await page.click('//button[3]')

// ❌ Text match without role (too broad)
await page.click('text="Continue"')

// ✅ Role + name (gold standard)
await page.getByRole('button', { name: 'Continue' })

// ✅ data-testid (when available)
await page.click('[data-testid="confirmButton"]')

// ✅ aria-label (stable, maintained by devs)
await page.click('[aria-label="Close dialog"]')

The API-First Mandate

Before navigating to any page to extract data, we ask: can I get this via HTTP directly? This single question has given us 60x speed improvements on read-only extraction tasks.

The decision tree:

• Public data — Try fetch() first. If you get a 200, skip the browser entirely.
• Auth-gated data — Arm an HTTP interceptor, navigate once to discover the private API, then use the API directly forever.
• Interactive/stateful — Use browser DOM extraction. This is the only case where you actually need the browser.

We arm HTTP interceptors on every navigation. When we discover a clean JSON API endpoint, we log it and use it directly on the next run. HackerNews, for example — we never scrape the page. We use the Firebase API directly. 60x faster.