Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News

This week on Ship It Weekly, LiteLLM's release pipeline was hijacked again: stolen GitHub CLI tokens from three engineers changed the pipeline, kicked off the release and approved it, and a credential-stealing 1.107.0 went out to PyPI and Docker Hub. GitHub put a package firewall into public preview, with advisory and supply chain checks and a cooldown for new releases. PHP fixed PHP2Shell, two critical core runtime bugs (CVE-2026-109618 and CVE-2026-109619) that someone tried to exploit four days before disclosure. And eMudhra, a certificate authority in India, found out the private keys for eight intermediate certificates could be worked out from its own public certificates.
The thread: most of this came through things you depend on but don't run. A library's release pipeline, the registry you pull from, the runtime under your app, and the trust store in your images.
In the lightning round: another GitLab CVSS 10 (CVE-2026-110264), Google's report on the europe-west2 London outage, AWS taking its European Sovereign Cloud off the global backbone on October 24, and a GitHub audit log change that can quietly break SIEM rules.
The human closer, from Brent Chapman: page on what users actually do. Your users don't care how your machines feel. They care whether they can log in.
On Monday, Jinger Meilani joins me to talk about running retros like a tabletop RPG.
Highlights
• LiteLLM 1.107.0: two stolen identities passed a two-person approval
• GitHub package firewall: advisories, attestations and a cooldown
• PHP2Shell: an exploit attempt four days before disclosure
• Patching PHP is easy, finding all of it is hard
• eMudhra: CA private keys worked out from public certificates
• Page on what users feel, not how the machine feels
Links
LiteLLM 1.107.0 compromise (Wiz): https://tsn.io/yQtmu
GitHub package firewall in public preview: https://tsn.io/QjofJ
PHP2Shell, CVE-2026-109618 and CVE-2026-109619 (Help Net Security): https://tsn.io/GUyop
Eight eMudhra CA keys compromised (The Hacker News): https://tsn.io/K2jqM
GitLab 19.5.1 patch release: https://tsn.io/0wsPX
Google Cloud europe-west2 incident report: https://tsn.io/nVk2N
AWS European Sovereign Cloud independent operation: https://tsn.io/gKnQl
GitHub secret scanning updates: https://tsn.io/JDGmT
Stop paging on how your machines feel (Brent Chapman): https://tsn.io/MAItM
SRE Weekly #539: https://tsn.io/XAsKz
This week's On Call Brief: https://tsn.io/nBAVg
More episodes and show notes: https://tsn.io/NqkdP
On Call Brief: https://tsn.io/Gpz2d
LMGT Awards: https://tsn.io/lAERA

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.
In this Ship It Conversations episode, I talk with Rob Koch about what accessible cloud native actually looks like, why sign language is still one of the hardest problems in AI, and why the fastest way to make your tooling accessible is to hire deaf engineers.
Rob is a senior principal at Slalom, an AWS Hero, and co-chair of the CNCF Deaf and Hard of Hearing Working Group. Rob is Deaf, and his ASL interpreter, Amelia Fruehsamer, voices his answers. Special thanks to Amelia for interpreting the conversation.
We start with an analogy Rob comes back to all episode: cloud native is the toolbox, and platform engineering is the workbench. Then we get into what's overhyped about AI, the guardrails it still needs, and what auditable AI observability looks like.
A big part of the conversation is accessibility in practice. Rob explains how captions pushed for deaf attendees ended up helping neurodivergent folks too, and what conferences get wrong: accessibility planned at the very end instead of the start, and the real cost of interpreters versus captions.
Then we get technical. Rob walks through using Kubernetes and Kubeflow to train models that recognize sign language, and why it's so hard: regional signs, lighting, skin tone and backgrounds, and a camera that can't tell which way a hand is facing. His joke: we don't have AGI until AI understands sign language.
The big takeaway: start with people, not a checklist. Hire deaf and hard of hearing engineers, ask them how they want to work, and your tools get more accessible because they're in the room.
Highlights
• Cloud native as the toolbox, platform engineering as the workbench
• What's overhyped about AI, and the guardrails it still needs
• What accessible cloud native means in practice
• Captions that help far more people than you planned for
• What conferences get wrong about accessibility, and what it costs
• Kubernetes and Kubeflow for sign language recognition
• Gesture-based computing and why speech-to-text had a 30-year head start
• Where to start: hire deaf engineers
Links
Rob Koch on LinkedIn: https://tsn.io/4qmFo
Rob Koch on GitHub: https://tsn.io/Mbxrs
Amelia Fruehsamer (Rob's ASL interpreter) on LinkedIn: https://tsn.io/UJB88
CNCF Deaf and Hard of Hearing Working Group: https://tsn.io/tB1lS
CNCF conference accessibility best practices: https://tsn.io/BZvpv
CNCF Merge Forward: https://tsn.io/QPVNQ
Kubeflow: https://tsn.io/4OVKf
More episodes and show notes: https://tsn.io/NqkdP
On Call Brief: https://tsn.io/Gpz2d
LMGT Awards: https://tsn.io/lAERA

This week on Ship It Weekly, GitHub Actions workflow lockfiles land in public preview, pinning actions and their dependencies to exact commits. GitLab shipped a critical patch release for CVE-2026-97229, a CVSS 10 CI/CD regex flaw that could let an unauthenticated attacker run code on the server. Google published its final report on two Cloud Storage incidents in us-central1, including a recurrence caused by an incomplete rollback. And Linus Torvalds pushed an experimental kernel branch with page-cache patches built using Claude Code, as maintainers debate dropping the mandatory AI attribution tag.
The thread this week: know what you're actually running, from the commit you reviewed to the state a rollback really left behind and code someone can still explain.
In the lightning round: a record Patch Tuesday with more than 1,400 CVEs and an actively exploited Windows Shell bypass, GitHub AI Scan for pull requests and agentic autofix going GA, AWS Strands Box, an open-source agent sandbox that keeps real credentials out of the agent's environment, and a Microsoft 365 outage through Global Secure Access.
The human closer comes from Paul LaPosta's “Your error budgets don't know AI exists.” AI has split shipping a change from understanding it, and SLOs don't measure comprehension. Your SLO can tell you the service is up. It cannot tell you whether anyone still understands it.
Programming note: Ship It Weekly turns one next month, so expect more conversations over the next couple of months, usually two a week, starting Monday with Rob Koch.
Links
GitHub Actions workflow lockfiles
GitLab 19.4.2 / 19.3.5 / 19.2.8 patch release
Google Cloud Storage incident report
Linus Torvalds and Claude Code (Phoronix)
Linux drops the Assisted-by requirement (Phoronix)
October 2026 Patch Tuesday
GitHub AI Scan for pull requests GA
Agentic autofix GA
AWS Strands Box
Microsoft 365 Global Secure Access outage
Your error budgets don't know AI exists (LeadDev)
This week's On Call Brief

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.
In this Ship It Conversations episode, I talk with Erik Osterman of Cloud Posse about Terraform modules, infrastructure as code frameworks, internal developer platforms, and why AI makes conventions more valuable, not less.
Erik is the founder and CEO of Cloud Posse, which has published more than 180 open source Terraform modules, and he hosts Cloud Posse's weekly DevOps Office Hours.
We start with what AI has done to open source infrastructure projects, then get into what makes a good Terraform module: consistent naming, a consistent interface, not so small that it wraps a single resource, and not so large that it has a hundred inputs nobody can test.
We also talk about CloudFormation, Pulumi, and why HCL's constraints helped build the module ecosystem. Erik explains where Crossplane fits, and why accounts, IAM, networking, DNS and certificates all come before the cluster.
A big part of the conversation is frameworks. Erik argues that nobody actually runs "vanilla Terraform," because Terraform alone doesn't handle state layout, promotion, authentication, tool installation or repository structure. Every team ends up building its own framework, usually undocumented. We talk about Terragrunt, Atmos, naming conventions, brownfield adoption, and breaking a monolithic root module into services.
Near the end, we get into AI and self-service. Erik's team now generates most of its infrastructure work with AI, and it works because the conventions, agent skills, MCPs and pipelines are already in place. Self-service doesn't have to mean another no-code platform.
The big takeaway: AI can generate infrastructure code faster than ever, but without conventions it is generating into chaos.
Highlights
• What makes a good Terraform module
• Root modules vs. child modules
• CloudFormation, Terraform, Pulumi, and why constraints helped HCL win
• Where Crossplane fits, and what has to exist before the cluster
• Why internal developer platforms like Backstage come after zero-to-one
• Why infrastructure as code needs frameworks, not just modules
• Agent skills, MCPs, and self-service without a no-code platform
Links
Cloud Posse: https://cloudposse.com
Atmos: https://atmos.tools
Cloud Posse on GitHub: https://github.com/cloudposse
DevOps Office Hours: https://tsn.io/2gq7v
Erik Osterman on LinkedIn: https://www.linkedin.com/in/osterman
More episodes and show notes: https://tsn.io/NqkdP
On Call Brief: https://tsn.io/Gpz2d
LMGT Awards: https://lmgt.org

This week on Ship It Weekly: Atlassian disclosed a critical arbitrary file access vulnerability affecting eight Data Center products, including Jira, Confluence, Bitbucket, and Bamboo. GitHub Copilot local sandboxing is now generally available, giving teams enforceable controls around filesystem, network, credentials, and local tooling. AWS published a deeper look at Lambda SnapStart for container images, including the operational risks of restoring initialized state. And Google Spanner Omni is now generally available, bringing Spanner outside Google Cloud and putting more of the availability responsibility back on the customer.
The bigger theme this week is control. Developer tools are production infrastructure. AI agents need real execution boundaries, not just instructions. Faster startup through snapshots means understanding exactly what state is being restored. And moving a managed database onto your own infrastructure gives you portability, but also gives you the pager.
In the lightning round: Citrix has a NetScaler SAML vulnerability under active exploitation. Cloudflare is working toward becoming a publicly trusted certificate authority. And attackers are actively scanning for Rejetto HFS servers vulnerable to a flaw that can lead to account takeover and remote code execution.
And the human closer comes from Lorin Hochstein and a recent Google Cloud networking incident. One part of the outage involved someone correctly following an incorrect procedure, while another involved someone incorrectly following a correct one. Training matters, but systems also need guardrails that keep normal human mistakes from turning into large outages. If your system only works when nobody makes a mistake, the system is the problem.
Links
Atlassian CVE-2026-21589
https://tsn.io/ZlqKv
GitHub Copilot Local Sandboxing
https://tsn.io/rQngM
AWS Lambda SnapStart for Container Images
https://tsn.io/uXqkK
Google Spanner Omni GA
https://tsn.io/GoZZ5
Citrix NetScaler CVE-2026-88779
https://tsn.io/yN5a9
Cloudflare Certificate Authority
https://tsn.io/egMN5
Rejetto HFS CVE-2026-61500
https://tsn.io/oQrkU
Google Cloud Networking Incident
https://www.tellerstech.com/go/s-dc989ad2/
There Was a Light That Never Went Out
https://www.tellerstech.com/go/s-cb0bc459/
This Week’s On Call Brief
https://tsn.io/8yaIY
Ship It Weekly
https://tsn.io/NqkdP
On Call Brief
https://tsn.io/Gpz2d