Move Inc. / Realtor.com · Experimentation at Scale

Testing as a decision system, not a button-color hobby.

Move.com and Realtor.com carried enormous traffic, which makes experimentation both possible and dangerous: you can learn fast, or you can generate a stream of statistically meaningless opinions. I helped build the program that made it the former.

Role: Senior Product Designer
Years: 2010 – 2011
Focus: High-traffic consumer · Experimentation · Mobile

Move.com multivariate testing program overview showing traffic, test details, performance metrics, and winning creative comparison

Context

Move Inc. operated Move.com, Realtor.com, and Mortgage Match, high-traffic consumer real estate products where small experience changes translated into meaningful revenue.

That traffic volume made rigorous experimentation feasible. It also made it necessary: with that many stakeholders and that much SEO and SEM investment, no one could win an argument by asserting taste.

I led UX across the mobile applications and partnered with Product, SEO, SEM, and Engineering to develop and execute the A/B and multivariate testing programs.

The challenge

User: House hunters arriving from search with wildly different intent, browsing, comparing, or ready to contact an agent, funneled through the same pages.
Business: Conversion to lead and contact drove revenue; incremental percentage points were material at that traffic level.
Organization: Product, SEO, SEM, design, and engineering each had legitimate and sometimes conflicting objectives for the same page.
Market: Competitive search-driven category where page changes carried SEO consequences alongside UX ones.

My mandate

Role: Senior Product Designer, mobile UX lead and experimentation partner.
Authority: Test design and UX direction in partnership with Product, SEO, SEM, and Engineering.
Team: Cross-functional partnership across product, engineering, SEO, SEM, analytics, and external development vendors. [ADD: team size]
Disciplines: UX, product, analytics, SEO/SEM, engineering, mobile development.
Stakeholders: Product management, business teams, engineering, external development vendors.
Timeframe: 2010 – 2011

What I learned first

Most 'losing' tests were badly framed, not badly designed.
Tests without a stated hypothesis and a primary metric produced results nobody could act on.
Traffic volume determines what is testable.
Multivariate designs were viable on the highest-traffic pages and wasteful further down the funnel, where sequential A/B tests answered faster.
SEO and UX conflicts were resolvable when made explicit.
Page structure decisions that looked like taste arguments were usually two teams optimizing different metrics on the same real estate.
Mobile was a different funnel, not a smaller one.
Intent, session length, and contact behavior differed enough across iPhone, iPad, Android, and Windows Mobile that desktop learnings did not transfer cleanly. [ADD: mobile-specific experiment results]

A test is not an opinion tiebreaker. It is a claim you were willing to be wrong about in public.

Strategy

Require a written hypothesis, primary metric, and expected effect size before a test entered the queue.
Prioritize experiments by traffic exposure, expected impact, and implementation cost rather than by who asked.
Match test design to available traffic, multivariate where volume supported it, sequential A/B where it did not.
Treat SEO, SEM, and UX as one experimentation program with shared measurement instead of parallel efforts on the same pages.
Close the loop: every result documented as a learning, whether it won, lost, or was inconclusive.

Decisions and tradeoffs

Gate experiments behind a written hypothesis and primary metric.

Why: Tests without a falsifiable claim generate data and no decisions, and they consume the same traffic as good tests.
Tradeoff: Slowed the queue and rejected ideas that stakeholders were attached to.
Result: A test backlog that produced actionable results and defensible decisions. [ADD: experiment volume and win rate]

Run experimentation jointly with SEO and SEM rather than as a UX activity.

Why: Those teams controlled traffic composition and page requirements. Testing without them produced wins that could not ship.
Tradeoff: More coordination overhead and slower test setup.
Result: Changes that survived both conversion and search objectives, contributing to improvements of up to 200% in conversion rates. [ADD: per-test breakdown]

Treat each mobile platform as its own UX problem.

Why: iPhone, iPad, Android, and Windows Mobile had different interaction conventions and different user contexts; a single ported design would underperform on all four.
Tradeoff: Higher design and engineering cost than one shared pattern.
Result: UX led across four platforms alongside Realtor.com and Mortgage Match contributions. [ADD: mobile conversion or engagement outcome]

Leading through it

Partnered with Product, SEO, SEM, and Engineering as one experimentation team with a shared backlog and shared metrics.
Turned stakeholder opinions into testable hypotheses, which resolved most disagreements without escalation.
Worked with external development vendors on mobile delivery, keeping design intent intact across handoffs.
Made results public inside the organization, including the losses, so the program built shared intuition instead of a scoreboard.

What we built

The experimentation program

Hypothesis development, prioritization, test design, traffic allocation, measurement, and iteration cycles across high-traffic pages and funnels.

[ADD: test matrix and experiment variant artifacts]

Mobile UX across four platforms

UX for iPhone, iPad, Android, and Windows Mobile applications, plus contributions to Realtor.com and Mortgage Match in partnership with product, engineering, business teams, and external development vendors.

[ADD: supporting image or artifact, mobile wireframes, before/after concepts]

Learning loops

Documented results and decisions so that a test outcome changed what the next test asked, rather than getting rerun by a new stakeholder six months later.

[ADD: analytics or decision framework artifact]

[ADD: supporting visuals, experiment variants, test matrices, wireframes, before/after concepts]

Outcomes

Up to 200%: conversion rate improvement from the testing program
4: mobile platforms with dedicated UX
[ADD]: experiments run and win rate
[ADD]: revenue impact

What changed

Page decisions moved from argument to evidence, which changed how the teams worked together as much as what shipped.
SEO, SEM, and UX stopped optimizing against each other on shared surfaces.
What I carry forward: an experimentation program is a decision system. The tests are the cheap part; the prioritization and the follow-through are the work.