Powered by RND
PodcastsEconomía y empresaLenny's Podcast: Product | Career | Growth

Lenny's Podcast: Product | Career | Growth

Lenny Rachitsky
Lenny's Podcast: Product | Career | Growth
Último episodio

Episodios disponibles

5 de 295
  • Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)
    Hamel Husain and Shreya Shankar teach the world’s most popular course on AI evals and have trained over 2,000 PMs and engineers (including many teams at OpenAI and Anthropic). In this conversation, they demystify the process of developing effective evals, walk through real examples, and share practical techniques that’ll help you improve your AI product.What you’ll learn:1. WTF evals are2. Why they’ve become the most important new skill for AI product builders3. A step-by-step walkthrough of how to create an effective eval4. A deep dive into error analysis, open coding, and axial coding5. Code-based evals vs. LLM-as-judge6. The most common pitfalls and how to avoid them7. Practical tips for implementing evals with minimal time investment (30 minutes per week after initial setup)8. Insight into the debate between “vibes” and systematic evals—Brought to you by:Fin—The #1 AI agent for customer serviceDscout—The UX platform to capture insights at every stage: from ideation to productionMercury—The art of simplified finances—Where to find Shreya Shankar• X: https://x.com/sh_reya• LinkedIn: https://www.linkedin.com/in/shrshnk/• Website: https://www.sh-reya.com/• Maven course: https://bit.ly/4myp27m—Where to find Hamel Husain• X: https://x.com/HamelHusain• LinkedIn: https://www.linkedin.com/in/hamelhusain/• Website: https://hamel.dev/• Maven course: https://bit.ly/4myp27m—In this episode, we cover:(00:00) Introduction to Hamel and Shreya(04:57) What are evals?(09:56) Demo: Examining real traces from a property management AI assistant(16:51) Writing notes on errors(23:54) Why LLMs can’t replace humans in the initial error analysis(25:16) The concept of a “benevolent dictator” in the eval process(28:07) Theoretical saturation: when to stop(31:39) Using axial codes to help categorize and synthesize error notes(44:39) The results(46:06) Building an LLM-as-judge to evaluate specific failure modes(48:31) The difference between code-based evals and LLM-as-judge(52:10) Example: LLM-as-judge(54:45) Testing your LLM judge against human judgment(01:00:51) Why evals are the new PRDs for AI products(01:05:09) How many evals you actually need(01:07:41) What comes after evals(01:09:57) The great evals debate(1:15:15) Why dogfooding isn’t enough for most AI products(01:18:23) OpenAI’s Statsig acquisition(1:23:02) The Claude Code controversy and the importance of context(01:24:13) Common misconceptions around evals(1:22:28) Tips and tricks for implementing evals effectively(1:30:37) The time investment(1:33:38) Overview of their comprehensive evals course(1:37:57) Lightning round and final thoughts—LLM Log Open Codes Analysis Prompt:Please analyze the following CSV file. There is a metadata field which has an nested field called z_note that contains open codes for analysis of LLM logs that we are conducting. Please extract all of the different open codes. From the _note field, propose 5-6 categories that we can create axial codes from.—Referenced:• Building eval systems that improve your AI product: https://www.lennysnewsletter.com/p/building-eval-systems-that-improve• Mercor: https://mercor.com/• Brendan Foody on LinkedIn: https://www.linkedin.com/in/brendan-foody-2995ab10b• Nurture Boss: https://nurtureboss.io/• Braintrust: https://www.braintrust.dev/• Andrew Ng on X: https://x.com/andrewyng• Carrying Out Error Analysis: https://www.youtube.com/watch?v=JoAxZsdw_3w• Julius AI: https://julius.ai/• Brendan Foody on X—“evals are the new PRDs”: https://x.com/BrendanFoody/status/1939764763485171948• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences: https://dl.acm.org/doi/abs/10.1145/3654777.3676450• Lenny’s post on X about evals: https://x.com/lennysan/status/1909636749103599729• Statsig: https://statsig.com/• Claude Code: https://www.anthropic.com/claude-code• Cursor: https://cursor.com/• Occam’s razor: https://en.wikipedia.org/wiki/Occam%27s_razor• Frozen: https://www.imdb.com/title/tt2294629/• The Wire on HBO: https://en.wikipedia.org/wiki/The_Wire—Recommended books:• Pachinko: https://www.amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935• Apple in China: The Capture of the World’s Greatest Company: https://www.amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373/• Machine Learning: https://www.amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955• Artificial Intelligence: A Modern Approach: https://www.amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133/Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].—Lenny may be an investor in the companies discussed.My biggest takeaways from this conversation: To hear more, visit www.lennysnewsletter.com
    --------  
    1:46:33
  • From managing people to managing AI: The leadership skills everyone needs now | Julie Zhuo (Facebook VP, Sundial CEO, The Making of a Manager author)
    Julie Zhuo is the former VP and Head of Design at Facebook (now Meta), author of the bestselling book The Making of a Manager, and co-founder of Sundial, an AI-powered data analysis company. Also, my first-ever podcast guest over 3 years ago!In our conversation, we discuss:1. The three core manager skills that translate directly to managing AI agents2. How her team uses AI to learn new skills 10x faster3. The “diagnose with data, treat with design” framework for balancing gut and data4. Why hypergrowth AI companies have terrible data infrastructure (and why it doesn’t matter)5. How to give feedback that actually lands—including Julie’s exact script for difficult conversations6. What Julie’s teaching her kids about an AI future (hint: it’s not coding or STEM)—Brought to you by:Mercury — The art of simplified financesDX — The developer intelligence platform designed by leading researchersPostHog—How developers build successful products—Transcript: https://www.lennysnewsletter.com/p/from-managing-people-to-managing-ai-julie-zhuo—My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/172723725/my-biggest-takeaways-from-this-conversation—Where to find Julie Zhuo:• X: https://x.com/joulee• LinkedIn: https://www.linkedin.com/in/julie-zhuo/• Website: https://www.juliezhuo.com/• Newsletter: https://lg.substack.com/• Sundial: https://sundial.so/—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, we cover:(00:00) Welcome back, Julie!(05:18) The success of The Making of a Manager(08:41) Why AI will make everyone a manager(11:38) The future of management roles(14:00) Empowering teams with AI(21:30) Specific roles being accelerated by AI(26:53) Data analysis in AI companies(32:02) The role of data in design(37:21) The evolving role of managers in the AI era(40:22) Embracing change and uncertainty(42:14) Timeless lessons for managers(49:03) Balancing strengths and weaknesses(57:49) Building a feedback culture(01:05:33) Creating win-win situations(01:09:27) Being aware of your own energy and conviction(01:12:12) Navigating disagreements with higher-ups(01:15:57) AI corner(01:20:08) Contrarian corner(01:23:14) Lightning round and final thoughts—Referenced:• Julie Zhuo on accelerating your career, impostor syndrome, writing, building product sense, using intuition vs. data, hiring designers, and moving into management: https://www.lennysnewsletter.com/p/episode-2-julie-zhuo• Waymo: https://waymo.com/• How we restructured Airtable’s entire org for AI | Howie Liu (co-founder and CEO): https://www.lennysnewsletter.com/p/how-we-restructured-airtables-entire-org-for-ai• Cursor: https://cursor.com/• The rise of Cursor: The $300M ARR AI tool that engineers can’t stop using | Michael Truell (co-founder and CEO): https://www.lennysnewsletter.com/p/the-rise-of-cursor-michael-truell• Inside ChatGPT: The fastest growing product in history | Nick Turley (Head of ChatGPT at OpenAI): https://www.lennysnewsletter.com/p/inside-chatgpt-nick-turley• Behind the founder: Marc Benioff: https://www.lennysnewsletter.com/p/behind-the-founder-marc-benioff• OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil (CPO at OpenAI, ex-Instagram, Twitter): https://www.lennysnewsletter.com/p/kevin-weil-open-ai• Anthropic’s CPO on what comes next | Mike Krieger (co-founder of Instagram): https://www.lennysnewsletter.com/p/anthropics-cpo-heres-what-comes-next• The Magic Loop: https://www.lennysnewsletter.com/p/the-magic-loop• Dunning-Kruger effect: https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect• Eric Antonow on LinkedIn: https://www.linkedin.com/in/antonow/• Methaphone: https://methaphone.com/• Replit: https://replit.com/• “Baby” by Justin Bieber on Spotify: https://open.spotify.com/track/6epn3r7S14KUqlReYr77hA• Kingdom Rush: https://www.kingdomrush.com/• Dr. Becky on TikTok: https://www.tiktok.com/@drbeckyatgoodinside• Emily Oster on TikTok: https://www.tiktok.com/@profemilyoster• La La Land on Netflix: https://www.netflix.com/title/80095365• Granola: https://www.granola.ai/• Matic robots: https://maticrobots.com/• Limitless pendant: https://www.limitless.ai/• How I AI: https://www.youtube.com/@howiaipodcast—Recommended books:• The Making of a Manager: What to Do when Everyone Looks to You: https://www.amazon.com/Making-Manager-What-Everyone-Looks/dp/0525540423• High Output Management: https://www.amazon.com/High-Output-Management-Andrew-Grove/dp/0679762884/• Zen and the Art of Motorcycle Maintenance: An Inquiry into Values: https://www.amazon.com/Zen-Art-Motorcycle-Maintenance-Inquiry/dp/0061673730• Conscious Business: How to Build Value Through Values: https://www.amazon.com/Conscious-Business-Build-through-Values/dp/1622032020• Good Inside: A Practical Guide to Resilient Parenting Prioritizing Connection Over Correction: https://www.amazon.com/Good-Inside-Guide-Becoming-Parent/dp/0063159481/—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected] may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com
    --------  
    1:36:24
  • Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor)
    Brendan Foody is the CEO and co-founder of Mercor, the fastest-growing company in history to go from $1M to $500M in revenue (in just 17 months!). At 22, he is also the youngest American unicorn founder ever. Mercor works with 6 of the Magnificent 7 and all top 5 AI labs to help them hire experts to create evaluations and training data that improve their models. In this conversation, Brendan explains why evals have become the critical bottleneck for AI progress, how he discovered this massive opportunity, and what the future of work might look like in an AI-driven economy.What you’ll learn:1. Why evals are becoming the primary bottleneck for AI progress and what this means for AI startups2. How Mercor grew to $500M revenue in 17 months (fastest in history)3. Brendan’s meeting with xAI that changed his company’s trajectory4. Which skills and jobs will remain most valuable as AI continues to advance (hint: jobs with “elastic” demand)5. Why Brendan believes AGI and superintelligence are not happening anytime soon6. The three unique core values that drove Mercor’s success7. How Harvard Lampoon writers are making Claude funnier—Brought to you by:WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUsJira Product Discovery—Atlassian’s new prioritization and roadmapping tool built for product teamsEnterpret—Transform customer feedback into product growth—Transcript: https://www.lennysnewsletter.com/p/experts-writing-ai-evals-brendan-foody—My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/173303790/my-biggest-takeaways-from-this-conversation—Where to find Brendan Foody:• X: https://x.com/BrendanFoody• LinkedIn: https://www.linkedin.com/in/brendan-foody-2995ab10b/—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, we cover:(00:00) Introduction to Brendan Foody and Mercor(05:38) The “era of evals”(09:26) Understanding the AI training landscape(17:10) The future of work and AI(25:54) The evolution of labor markets(29:55) Understanding how AI models are trained(38:58) Building Mercor(53:27) Lessons from past ventures(56:55) The future of AI and model improvement(01:00:41) His personal use of AI and final thoughts—References: https://www.lennysnewsletter.com/p/experts-writing-ai-evals-brendan-foody—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected] may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com
    --------  
    1:07:08
  • The ultimate guide to AEO: How to get ChatGPT to recommend your product | Ethan Smith (Graphite)
    Ethan Smith is the CEO of Graphite—the leading SEO growth agency—and my go-to expert on SEO. After 18 years of mastering traditional SEO, Ethan has been at the forefront of what is called AEO: answer engine optimization, or, more simply, getting your product to show up in ChatGPT/Claude/Gemini/Perplexity answers. He’s discovered that ChatGPT traffic converts six times better than Google search—and most companies are completely missing this opportunity.In our conversation, we discuss:1. His 7-step playbook to rank #1 in ChatGPT2. Why ChatGPT traffic converts 6x better than Google3. How early-stage startups can win at AEO immediately (unlike with SEO, which takes years)4. The three tactics that actually work: landing pages, YouTube videos, and Reddit comments5. Why help-center content can suddenly be your highest-ROI investment6. The specific Reddit strategy that works (spoiler: be authentic)7. Why AI-generated content doesn’t work—Brought to you by:Orkes—The enterprise platform for reliable applications and agentic workflowsVanta—Automate compliance. Simplify security.Great Question—Empower everyone to run great research—Where to find Ethan Smith:• Twitter: https://twitter.com/ethan_l_s• LinkedIn: https://bit.ly/ethans-linkedin• Graphite: https://graphite.io/• Graphite Research Papers: https://bit.ly/graphite-five-percent—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, we cover:(00:00) Welcome back, Ethan(04:34) The changing landscape of SEO(06:19) AEO (answer engine optimization) vs. GEO (generative engine optimization)(08:13) The impact of AEO(11:51) How early-stage startups can win at AEO(14:34) The quality of AEO leads(15:35) On-site vs. off-site traffic(16:32) Reddit’s role in AEO and avoiding spam(20:11) How AI models use citations (RAG)(21:41) Key principles for winning at AEO(25:00) Avoiding hyper-SEOed content, and the importance of originality(28:55) Actionable AEO playbook: steps and experiments(33:35) Tracking, measuring, and share of voice(38:34) Adapting AEO for B2B, commerce, and early-stage companies(41:11) Is letting AI index your content good?(43:06) Experimentation, control groups, and measuring results(46:15) The future of AEO, SEO, and search channels(51:35) AI-generated content: what works and what doesn’t(55:25) The dangers of infinite AI derivatives(58:44) The future: convergence of LLMs and search(01:00:40) Help-center optimization and the long tail(01:03:18) Lightning round and final thoughts—Resources and episode mentions: https://www.lennysnewsletter.com/p/the-ultimate-guide-to-aeo-ethan-smith—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].—Lenny may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com
    --------  
    1:11:55
  • $46B of hard truths from Ben Horowitz: Why founders fail and why you need to run toward fear (a16z co-founder)
    Ben Horowitz is the co-founder of Andreessen Horowitz, Silicon Valley’s largest and most influential venture capital firm, with over $46B in committed capital across multiple funds. He took Loudcloud public with just $2 million in revenue (dubbed “the IPO from hell”), sold it for $1.6 billion, and has backed companies from Facebook to Stripe to Airbnb to OpenAI to Databricks (now worth more than $100 billion). His management philosophy—forged through near-death experiences and refined through coaching hundreds of CEOs—contradicts most conventional startup wisdom.In our conversation, Ben shares:1. Why “founder mode” is half right and half dangerously wrong2. The story behind “Good Product Manager/Bad Product Manager” and why it went viral despite being written in anger3. Where the biggest AI startup opportunities remain4. Why you need to run toward fear, never away5. The one trait that predicts that a founder will fail as CEO6. Inside Paid in Full, Ben’s nonprofit awarding pensions to pioneering hip-hop artists—Brought to you by:DX—The developer intelligence platform designed by leading researchers: http://getdx.com/lennyBasecamp—The famously straightforward project management system from 37signals: https://www.basecamp.com/lennyMiro—A collaborative visual platform where your best work comes to life: https://miro.com/lenny—Transcript: https://www.lennysnewsletter.com/p/46b-of-hard-truths-from-ben-horowitz—My biggest takeaways (for paid newsletter subscribers): ⁠https://www.lennysnewsletter.com/i/172439345/my-biggest-takeaways-from-this-conversation—Where to find Ben Horowitz:• X: https://x.com/bhorowitz• LinkedIn: https://www.linkedin.com/in/behorowitz/• Website: https://benhorowitz.com/• Andreessen Horowitz’s website: https://a16z.com/—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, we cover:(00:00) Introduction to Ben Horowitz(04:09) Important leadership lessons from Shaka Senghor(10:15) Running toward fear and why hesitation kills companies(19:35) Who shouldn’t start a company(22:36) The Databricks story: thinking bigger(24:54) Managerial leverage and CEO psychology(28:06) When founders should be replaced as CEOs(31:20) Normalizing failure for CEOs(37:57) Counterintuitive lessons about building companies(42:31) “Good Product Manager/Bad Product Manager”(48:21) Product managers as leaders(51:16) Why a16z invested in Adam Neumann after WeWork(56:23) Is AI in a bubble?(01:02:43) The biggest opportunities in AI(01:12:51) Why U.S. leadership in AI matters(01:18:53) The Paid in Full Foundation for hip-hop pioneers(01:23:18) Lightning round: book recommendations, products, and life mottos—References: https://www.lennysnewsletter.com/p/46b-of-hard-truths-from-ben-horowitz—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected] may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com
    --------  
    1:37:59

Más podcasts de Economía y empresa

Acerca de Lenny's Podcast: Product | Career | Growth

Interviews with world-class product leaders and growth experts to uncover concrete, actionable, and tactical advice to help you build, launch, and grow your own product.
Sitio web del podcast

Escucha Lenny's Podcast: Product | Career | Growth, Tengo un Plan y muchos más podcasts de todo el mundo con la aplicación de radio.net

Descarga la app gratuita: radio.net

  • Añadir radios y podcasts a favoritos
  • Transmisión por Wi-Fi y Bluetooth
  • Carplay & Android Auto compatible
  • Muchas otras funciones de la app

Lenny's Podcast: Product | Career | Growth: Podcasts del grupo

Aplicaciones
Redes sociales
v7.23.9 | © 2007-2025 radio.de GmbH
Generated: 9/26/2025 - 6:44:55 AM