Experiment Nation: The Podcast

Starting a new CRO or experimentation role can be exciting—but one of the biggest mistakes is rushing to run tests before you understand the business.In this episode of the Experiment Nation Podcast, Ross Newman speaks with Louisa about what really happens when you move into a new organization or even an entirely new industry.They discuss:• What to do in your first 30 days in a new CRO role • Why listening to stakeholders may be more important than launching tests • How to assess an organization's experimentation maturity • The difference between a company that *says* it experiments and one where experimentation is genuinely part of the culture • Red flags to look for when interviewing for a CRO role • Why CRO professionals shouldn't try to do everything themselves • Where the boundaries of a CRO team's responsibilities should be • How analytics, research and experimentation fit together • How to build credibility when you're the new person in the organization • Whether experimentation really differs between industries • How AI and LLMs could fundamentally change experimentation • The future of experimentation beyond traditional websites and apps • Why Ross moved from product management into CRO • And whether CRO should actually sit within productLouisa also shares an important lesson from moving across multiple industries: while organizations may look very different on the surface, the fundamental principles of experimentation, consumer psychology, research and understanding customers remain remarkably consistent.If you're starting a new CRO role, moving from agency to in-house, changing industries, or trying to build an experimentation culture from scratch, this conversation is packed with practical lessons.Time Chapter00:00 Introduction01:18 Louisa's Journey Into CRO03:53 Why Analytics Is the Foundation of Experimentation05:00 Your First 30 Days in a New CRO Role07:22 Junior vs. Senior Experimenters08:08 Does the Company Really Have an Experimentation Culture?12:03 Does Company Size Change How You Experiment?14:01 Educating Senior Leadership About CRO16:02 The Biggest Mistakes When Starting a New CRO Role18:54 You Can't Do CRO Alone21:20 What Actually Belongs in a CRO Role?25:56 How AI Is Changing CRO28:57 How to Build Credibility in a New Organization32:02 How to Improve an Existing Experimentation Process34:00 Does CRO Really Differ Between Industries?39:08 Experimentation Beyond the Website41:04 Can We Experiment Inside LLMs?44:08 Why Ross Moved From Product to CRO49:38 Should CRO Sit Within Product?52:51 The Fundamentals of Experimentation53:13 CRO Meets Formula 154:08 Final Thoughts#CRO #Experimentation #A_BTesting #ConversionRateOptimization #ProductManagement #growth Subscribe to Experiment Nation for more conversations about CRO, experimentation, A/B testing, growth, and product experimentation.---Music: “Strong Female Pop Attitude” by JuicyLucyMusic, licensed via AudioJungle/Envato Market.https://audiojungle.net/item/strong-pop-attitude/19345362


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

High-velocity experimentation can dramatically increase how quickly organizations learn, optimize their products, and drive revenue. But many teams still run A/B tests sequentially, waiting for one experiment to finish before starting the next.
In this video, Varun, co-founder of Geeks of Digital, explains how overlapping A/B tests can remove experimentation bottlenecks while still maintaining rigorous randomization and measurement.
You’ll learn how layered experimentation works, why the "isolation fallacy" can hold teams back, when experiments should still be isolated, and how organizations can build an operational system for running experiments at scale.
The video also covers the "blue-by-blue" isolation trap, compounding gains from multiple winning experiments, fake door testing, experimentation culture, and how companies like Meta, Booking.com, Airbnb, and Spotify approach experimentation.
---
Music: “Strong Female Pop Attitude” by JuicyLucyMusic, licensed via AudioJungle/Envato Market.https://audiojungle.net/item/strong-pop-attitude/19345362


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Starting a career in Conversion Rate Optimization (CRO) can be surprisingly difficult. Getting your first experience, working with developers, prioritizing experiments, dealing with failed tests, and proving the value of experimentation all come with a learning curve.In this episode of Experiment Nation, Eline speaks with CRO and experimentation leader Edd about the lessons he learned during his first years in the industry, and what he wishes he had known sooner.They discuss:• How to get your first CRO experience when you don't have a CRO background• Why agency experience can be a powerful accelerator for learning CRO• Getting buy-in from developers and explaining experimentation to technical teams• Why traditional prioritization frameworks like ICE and P.I.E. don't always work• How to prioritize experiments using potential business impact and MDE• How to build a CRO career from digital marketing, analytics, UX, or other disciplines• Why your first experiments don't have to be perfect A/B tests• How to deal with failed and inconclusive experiments• Why CROs should spend more time in the problem space and less time generating test ideas• Using quantitative and qualitative research to identify customer problems• Jobs to Be Done, surveys, stakeholder interviews, and customer journey mapping• Why experimentation should generate customer intelligence, not just conversion lifts• How previous experiment results can inform your next hypothesis• Dealing with imposter syndrome and the fear of failure• Why CRO professionals should "have strong opinions, but hold them loosely"• The importance of learning from mistakes and communicating them effectivelyOne of the biggest themes of the conversation is the shift from "What should we test?" to "What problem are our customers actually experiencing?"For CRO practitioners, that's an important distinction. The goal isn't simply to generate more winning experiments. It's to understand customers better, validate problems, and build a growing body of evidence that makes future experimentation more effective.Whether you're starting your first CRO role or you've been running experiments for years, there are plenty of practical lessons here.Chapters:00:00 Introduction01:03 Meet Ed: 10+ Years in CRO & Experimentation02:26 An Unusual Path Into Digital Marketing05:22 Discovering Experimentation Through Email Testing08:34 How Previous Careers Can Prepare You for CRO10:20 How to Get Your First CRO Experience12:28 CRO Communities, Courses & Learning Resources15:42 Why Practical Experience Beats Theory16:00 The Hardest Parts of Starting a CRO Career16:24 Getting Technical Buy-In From Developers17:30 Why CRO Prioritization Frameworks Don't Always Work18:32 How to Prioritize Experiments Based on Business Impact21:38 Getting Your First CRO Client or Project23:01 How to Create CRO Opportunities In-House24:29 Why Agency Experience Can Accelerate Your CRO Career27:19 Turning Digital Marketing Work Into CRO Experience28:21 Experimenting Beyond A/B Tests30:08 Learning From Failed Experiments32:00 Overcoming Imposter Syndrome & Fear of Failure34:41 How to Communicate Mistakes at Work37:29 Get Out of the Solution Space38:43 Finding & Validating Customer Problems40:25 Stop Chasing Wins, Start Chasing Learnings45:14 Using Jobs to Be Done in CRO46:42 Surveys & Stakeholder Interviews for Customer Research47:24 Customer Journey Mapping for CRO50:40 Don't Forget the Human Behind the Data51:57 "Have Strong Opinions, But Hold Them Loosely"53:41 Keep Learning Beyond Traditional CRO Resources54:29 What CRO Can Learn From Radical Candor56:04 Final Thoughts#CRO #ConversionRateOptimization #Experimentation #ABTesting #GrowthMarketing #UX #DigitalMarketingBackground music:"Background Hip-Hop Urban" by CoffeeMusicLicensed via AudioJungle under the Music Standard License.


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Jon Crowder joins Tracy Laranjo to explain why ethical experimentation is not just better for customers—it is better for long-term growth. They unpack fake urgency, misleading pricing and subscription tactics, short-term A/B test wins that erode trust, and practical ways to design experiments that improve retention, lifetime value, and revenue.Chapters:0:00 Welcome & introducing Jon Crowder1:29 Why ethical experimentation matters2:17 The pressure to drive results at any cost7:17 When winning an A/B test loses money long term8:34 Reframing urgency into trust-building experiences13:38 Choosing clients and setting ethical boundaries16:37 The ethics of price testing19:21 Fake sales, subscriptions & agency pressure20:46 How to steer unethical tests into ethical ones23:09 Have urgency and scarcity gone too far?24:02 Why legacy businesses cling to outdated tactics27:36 Gen Z, trust, and changing expectations29:48 What an ethical future for digital experiences looks like34:30 Brands that earn loyalty through values38:27 Why ethical experiences convert better40:15 A simple test for ethical experimentation41:37 Where to find Jon#CRO #Experimentation #ABTesting #GrowthMarketing #EthicalMarketing #ProductGrowth #ConversionOptimization #DigitalExperience #CustomerExperience #Retention #MarketingStrategy #GrowthStrategy


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode, we dive into CRO “best practices” that completely failed in real-world experimentation. From homepage carousels unexpectedly winning tests to why most experimenters still don’t trust AI-generated CRO workflows, we unpack the messy reality behind optimization advice.We also debate synthetic AI users for research, discuss the most useful survey questions in CRO, roast dark-mode websites, and explain why emotional branding can sometimes hurt usability.Join our slack: https://join.slack.com/t/experiment-nation/shared_invite/zt-3zowejvoj-ujS_tbqfXcPkS9O0rnWhTQTopics include:Why homepage carousels occasionally winAI-powered A/B testing platformsSynthetic users & fake research dataBest and worst CRO survey questionsDark mode website UXWhy qualitative context still mattersJoin the Experiment Nation Slack community to participate in future discussions and question threads.Chapters00:00 Intro & CRO community questions01:46 CRO best practices that failed in real life02:28 Do homepage carousels actually work?05:02 Hero videos, agencies & emotional design06:25 When carousels might make sense08:24 Should AI run the entire CRO workflow?11:48 Why experimenters still don’t trust AI14:49 Best & worst survey questions in CRO17:30 Emotional branding vs usable websites18:26 Dark mode websites & readability debate21:29 Synthetic AI users for testing & research23:56 Why fake users can’t mimic real humans26:32 Using AI as a research starting point28:41 Closing thoughts & Slack community invite


This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com