This article explains how to design, run, and evaluate incrementality tests in your affiliate program. It covers how to define incrementality, choose the right metrics, build a testing plan, and control variables to ensure reliable results. Testing helps you understand the additional value your program generates and supports better optimization decisions.
Define incrementality
Start by agreeing on what incremental value means for your business. This should be aligned with your marketing and commercial goals. Examples include growing basket values, acquiring new customers, or increasing sales in priority product categories.
When defining incrementality, think about:
The challenges you want to solve.
Business areas where additional value matters most.
The role different partner types play in your customer journey.
This definition becomes the foundation for your testing plan.
Identify the right metrics
Choose metrics that match your definition of incremental value. Different goals require different measurements.
Examples include:
Basket boost, if your priority is increasing order values.
New customer rate, if you’re focused on customer acquisition.
Initiator ratio, if you want to understand who starts your purchase journeys.
The metrics you choose will determine how you measure success during your tests.
Build a testing plan
An effective testing plan allows you to understand the true value of partners, placements, and promotions. To build one, follow these steps.
Identify partners: Decide which partner types you want to test and what incremental value you expect each type to deliver. This helps you structure your test around clear objectives.
Establish the testing methodology: Choose a method that fits your partner type and goals. Examples include:
Placement testing, where you keep the offer the same but test different placements.
Offer testing, where you keep the placement the same but vary the promotion.
BAU vs. test periods, allowing you to compare uplift.
Partner hold-outs or controlled exclusion tests.
Always use the same testing method across similar partner types to maintain consistency.
Set test periods and BAU periods: Define clear timeframes for testing and compare them with business-as-usual performance. This helps you assess the strategic impact of your changes.
Test placements: Test different placements where relevant. Keep the promotion constant so you can measure the effect of the placement alone on your chosen incremental metric.
Test promotions or offers: Once you have identified your strongest placements, test different offers within those placements. This helps you understand which promotions drive the most incremental value.
Review regularly: Revisit your testing results over time. Conditions change, so testing periodically ensures your strategy remains effective.
Control the variables
Accurate incrementality testing depends on controlling as many variables as possible. There are factors you can control and factors you can’t. Your testing plan should account for both.
Variables you can control
On-site offers: Keep your on-site proposition consistent during tests. For example, a 20% site-wide promotion will impact the performance of a 10% influencer offer.
Placements and promotions: Change one variable at a time. Keep promotions the same when testing placements, and keep placements the same when testing promotions.
Variables outside your control
Seasonal variation: Peak trading, payday periods, or seasonal events can impact performance.
Competitor activity: Competitor offers may directly influence your partners’ ability to drive results.
Macro‑economic conditions: Confidence may be impacted by interest rates or economic forecasts.
Weather conditions: Events like heatwaves can reduce e‑commerce demand. Research shows an average 30% drop in sales during hot summer periods.
Since these factors can affect your results, run multiple tests to smooth out variability.
Implement your learnings
Once your tests are complete, review the outcomes and apply the learnings to your ongoing strategy.
Use this insight to:
Adjust commissioning or incentive structures.
Optimize placements and promotions.
Scale activity with high-value partners.
Improve your long-term approach to driving incremental value.
Incrementality should not be a one-off exercise. Revisit your testing approach regularly so that it aligns with your evolving business goals.
FAQ
How often should I run incrementality tests?
Tests should be repeated at regular intervals. Factors like seasonality, competitor activity, and economic changes can influence performance, so repeated testing helps ensure your findings remain accurate.
Should I test placements and offers at the same time?
No. To understand true uplift, change only one variable at a time. For example, keep the offer consistent when testing placements.
Do blackout tests help determine incrementality?
Blackout tests can create distortions and risk lost revenue or market share. They should be used with caution and only as part of a well-designed testing framework.