Ally Financial Careers

At Ally, you get a startup feel, but experience the benefits of a company that has worked out the kinks and is fulfilling its purpose. We are always evolving and see that as a good thing. From owning our work to seeing its impact in the real world, our team is relentless in finding new ways technology can help make experiences better and help people. We are problem solvers, we value diverse thinking, we support one another, and we challenge ourselves to think bigger in the journey to deliver customer-obsessed tech solutions. To read more about what our tech team does, be sure to visit our tech blog at ally.tech

You will bring SRE practice to the AI agent ecosystem — defining and enforcing production readiness standards, building the observability and alerting layer, running the readiness gate for every service before it goes live, and owning the incident response process when things fail. You will also build and operate the SRE agents themselves: the automated tools that run production readiness checks, generate post-incident reviews, monitor SLO burn rates, and surface reliability findings before a deployment proceeds.

This is not a traditional SRE role watching dashboards. You are an active builder. The SRE toolchain here is itself an agent-driven system — you will extend it, maintain its knowledge core content, and use it to enforce standards across every team in the program.

At this time, Ally will not sponsor a new applicant for employment authorization for this position.

Production readiness and SLO ownership

• Run the 10-point production readiness gate for every Lightspeed and Logos service before first production deploy — SLOs defined, runbook exists, alerting configured, rollback documented, on-call assigned
• Define and maintain Dynatrace SLOs for AI-powered services; configure burn-rate alerting (multi-window, aligned to user impact)
• Own the error budget policy: track consumption, flag services approaching exhaustion, enforce the deployment freeze when budgets are gone

Observability for AI workloads

• Instrument AI agent pipelines with structured JSON logging (traceId, spanId, correlationId), custom metrics, and distributed traces
• Build Dynatrace dashboards for AI services: request rate, error rate, latency P50/P95/P99, dependency health, agent invocation counts and failure rates
• Identify and address the observability gaps that make AI system failures hard to diagnose — context truncation, tool call failures, model timeouts, partial completions

SRE agent development and maintenance

• Own the sre-gate, sre-monitor, sre-pir, sre-remediation, and sre-validation agents — keep their behavioral rules, domain context, and integration patterns current
• Maintain the SLO definitions, domain team map, incident classification, and runbook location config in the knowledge core (domain.md, architecture.md)
• Extend the SRE reliability anti-pattern checklist as new failure modes are identified in AI-powered services

Incident management and PIR

• Lead post-incident reviews for production AI service failures; generate structured PIRs and track remediation items through to closure in Jira
• Monitor for recurring patterns (same root cause three or more times triggers systemic pattern review); drive elimination of systemic issues
• Maintain on-call rotation coverage across the service portfolio; run escalation path validation before each new service goes live

Reliability design review

• Review PRs and architecture designs for AI services against the reliability anti-pattern checklist before merge: retries, circuit breakers, health endpoints, graceful degradation, rollback plans
• Issue BLOCKING vs ADVISORY findings with specific file and line citations
• Work with developers to resolve BLOCKING findings before they reach production

Minimum Requirements:

• 3+ years of experience
• High School Diploma or GED equivalent

Preferred Requirements:

• 4+ years in SRE, platform engineering, or production operations with direct ownership of production services
• Deep familiarity with SLO/SLI/error budget concepts — not just the theory, but day-to-day operation in a real environment
• Experience with an observability platform at the level of configuring alerts and dashboards, not just reading them (Dynatrace, Datadog, New Relic, or equivalent)
• Strong Python or TypeScript — you write and own automation, not just runbooks

Strongly preferred

• Experience instrumenting distributed systems for observability: structured logging, metrics, distributed tracing
• Familiarity with the Anthropic Claude API or Claude Code — understanding what makes AI agent systems fail differently from conventional services
• Experience running post-incident reviews and tracking systemic remediation to completion
• Confluence and Jira integration experience — the SRE toolchain here publishes directly to both
• Financial services or other highly regulated environment experience

What sets the best candidates apart

• You have written the alerting rules and SLO definitions, not just responded to the pages they generate
• You understand the specific failure modes of AI systems: non-determinism, context limits, tool call failures, latency spikes from model inference — and you know how to build observability that surfaces them
• You treat reliability as a shared engineering discipline, not an ops function — you work with developers before code merges, not after pages fire
• You have maintained runbooks that were actually used during incidents, which means they were accurate, concise, and kept current