Site Reliability Engineer

Remote

GitLab logo
GitLab
Apply now Apply later

Posted 3 weeks ago

Site Reliability Engineers (SREs) are responsible for keeping all user-facing services and other GitLab production systems running smoothly. SREs are a blend of pragmatic operators and software craftspeople that apply sound engineering principles, operational discipline, and mature automation to our environments and the GitLab codebase. We specialize in systems, whether it be networking, the Linux kernel, or some more specific interest in scaling, algorithms, or distributed systems.

GitLab.com is a unique site and it brings unique challenges–it’s the biggest GitLab instance in existence. In fact, it’s one of the largest single-tenancy open-source SaaS sites on the internet. The experience of our team feeds back into other engineering groups within the company, as well as to GitLab customers running self-managed installations.

  • Be on a PagerDuty rotation to respond to GitLab.com availability incidents and provide support for service engineers with customer incidents.
  • Use your on-call shift to prevent incidents from ever happening.
  • Run our infrastructure with Chef, Terraform and Kubernetes.
  • Make monitoring and alerting alert on symptoms and not on outages.
  • Document every action so your findings turn into repeatable actions–and then into automation.
  • Use the GitLab product to run GitLab.com as a first resort and improve the product as much as possible
  • Improve the deployment process to make it as boring as possible.
  • Design, build and maintain core infrastructure pieces that allow GitLab scaling to support hundred of thousands of concurrent users.
  • Debug production issues across services and levels of the stack.
  • Plan the growth of GitLab's infrastructure.
  • Think about systems - edge cases, failure modes, behaviors, specific implementations.
  • Know your way around Linux and the Unix Shell.
  • Know what is the use of config management systems like Chef (the one we use)
  • Have strong programming skills - Ruby and/or Go
  • Have an urge to collaborate and communicate asynchronously.
  • Have an urge to document all the things so you don't need to learn the same thing twice.
  • Have an enthusiastic, go-for-it attitude. When you see something broken, you can't help but fix it.
  • Have an urge for delivering quickly and iterating fast.
  • Share our values, and work in accordance with those values.
  • Have experience with Nginx, HAProxy, Docker, Kubernetes, Terraform, or similar technologies
  • Ability to use GitLab
  • Coding infrastructure automation with Chef and Terraform
  • Improving our Prometheus Monitoring or building new Metrics
  • Helping release managers deploy and fix new versions of GitLab-EE.
  • Plan, prepare for, and execute the migration of GitLab.com from virtual machines running on Google Cloud to cloud-native container-based deployments with Kubernetes using Google Kubernetes Engine.
  • Develop a relationship with a product group, define their SLAs, share GitLab.com data on those SLAs and improve their reliability
Leveling of Site Reliability Engineering at GitLab
  • Use Chef and Ansible to efficiently manage our infrastructure
  • Implement Infrastructure as Code using Terraform and GitLab CI/CD for automation
  • Load balancing the application including Proxies and CDN
  • Kubernetes and containerizing our system
  • Administer a high-availability PostgreSQL cluster.
  • Monitoring and Metrics in Prometheus, Grafana and integrations with Slack/PagerDuty
  • Logging infrastructure
  • Backend storage management and scaling
  • Disaster Recovery and High Availability strategy
  • Contributing to code in GitLab
  • Team organization and planning
  • Issue, Epic, OKR leadership and completion
  • Creating blog posts
  • Completing Root Cause Analysis (RCA) investigations
  • Contributions to handbook, runbooks, general documentation
  • Leading and contributing to designs for issues, epics, okrs
  • Improving team practices in handoffs of work and incidents
  • Involvement in hiring process - reviewing questionnaires, involved in interviews, qualifying candidates
  • Knowledge sharing, mentoring
  • Accountability, Self awareness, handling conflict in the team and receiving feedback
  • Maintaining good relationships with other engineering teams in GitLab that help improve the product

The Junior Site Reliability Engineer is a grade 5.

Technical:

  1. Updates GitLab default values so there is no need for configuration by customers.
  2. General knowledge of the 2 of the areas of technical expertise

Execution:

  1. Provides emergency response either by being on-call or by reacting to symptoms according to monitoring and escalation when needed
  2. Delivers production solutions that scale, identifies automation points, and proposes ideas on how to improve efficiency.
  3. Improves monitoring and alerting fighting alert spam.

Collaboration and Communication:

  1. Improves documentation all around, either in application documentation, or in runbooks, explaining the why, not stopping with the what.

Influence and Maturity

  1. Shares the learnings publicly, either by creating issues that provide context for anyone to understand it or by writing blog posts.

The Site Reliability Engineer is a grade 6.

Technical:

  1. General knowledge of the 4 of the areas of technical expertise with deep knowledge in 1 area

Execution:

  1. Provides emergency response either by being on-call or by reacting to symptoms according to monitoring and escalation when needed
  2. Proposes ideas and solutions within the infrastructure team to reduce the workload by automation.
  3. Plan, design and execute solutions within infrastructure team to reach specific goals agreed within the team.
  4. Plan and execute configuration change operations both at the application and the infrastructure level.
  5. Actively looks for opportunities to improve the availability and performance of the system by applying the learnings from monitoring and observation

Collaboration and Communication:

  1. Improves documentation all around, either in application documentation, or in runbooks, explaining the why, not stopping with the what.

Influence and Maturity

  1. Shares the learnings publicly, either by creating issues that provide context for anyone to understand it or by writing blog posts.
  2. Contributes to the hiring process in review questionnaires or being part of the interview team to qualify SRE candidates

The Senior Site Reliability Engineer is a grade 7.

Are experienced Site Reliability Engineers who meet the following criteria

Technical:

  1. Deep knowledge in 2 areas of expertise and general knowledge of all areas of expertise. Capable of mentoring Junior in all areas and other SRE in their area of deep knowledge.
  2. Contributes small improvements to the GitLab codebase to resolve issues

Execution:

  1. Identifies significant projects that result in substantial cost savings or revenue
  2. Identifies changes for the product architecture from the reliability, performance and availability perspective with a data driven approach.
  3. Proactively work on the efficiency and capacity planning to set clear requirements and reduce the system resources usage to make GitLab cheaper to run for all our customers.
  4. Identify parts of the system that do not scale, provides immediate palliative measures and drives long term resolution of these incidents.
  5. Identify Service Level Indicators (SLIs) that will align the team to meet the availability and latency objectives.

Collaboration and Communication:

  1. Know a domain really well and radiate that knowledge through recorded demos, discussions in DNA meetings, or Incident Reviews
  2. Perform and run blameless RCAs on incidents and outages aggressively looking for answers that will prevent the incident from ever happening again.

Influence and Maturity:

  1. Set an example for team of SREs with positive and inclusive leadership and discussion on work.
  2. Show ownership of a major part of the infrastructure.
  3. Trusted to de-escalate conflicts inside the team

Your Privacy

For information about our privacy practices in the recruitment process, please visit our Recruitment Privacy Policy.

 
Job tags: Ansible CD Chef CI Docker Gitlab Go Grafana High availability Kubernetes Linux Load Balancing Nginx PostgreSQL Prometheus Reliability engineering Ruby Terraform Unix
Job region(s): Remote/Anywhere
Job stats:  416  15  1
  • Share this job via
  • or