Definition
The Senior NoSQL Data Engineer oversees junior data engineering activities and aiding in building the business’ data collection systems and processing pipelines. Senior Data Engineer is responsible for building and maintaining optimized and highly available data pipelines that facilitate deeper analysis and reporting by the Data and Analytics department.
The Senior Data Engineer strives to develop new and improved data engineering capabilities continuously. The Senior Data Engineer builds data processing frameworks that handle the business’s growing database. He works with senior data science leadership and other Data and Analytics teams to leverage data with reporting and scientific tools, such as Tableau, R, and Spark.
Daily Responsibilities
- Designing and developing data models to create new databases or update existing ones.
- Collaborate across teams on the design and maintenance of our Operations data mart (ETL, data modeling, metric design, reporting/dashboarding) to ensure a stable reporting infrastructure.
- Lead investigations streams of content usage and A/B testing.
- Take on research projects to improve data processing and any implemented machine learning frameworks used by our team.
- Be exemplary in promoting Data Engineering best practices within the team.
- Introduce new skills and knowledge into the team, adding to the already dynamic environment our Data Engineering team has created.
- Providing consultation on data management issues to other members of the Salla staff.
- Share knowledge to complement team-wide expertise.
- Develop high-quality software at a senior level.
- Developing solutions that can be implemented in a production environment to improve an organization’s data management practices
- Assist with the elicitation and documentation of system requirements.
- Create conceptual architectures and detailed designs for software solutions.
- Working with data architects to design enterprise data warehouses for storing large quantities of data for later retrieval.
- Understand business and technical requirements and constraints to design effective software solutions.
- Recommending changes to existing databases to improve performance or resolve issues.
- Developing reports that present data in an easy-to-understand format for nontechnical users such as managers.
- Using statistical analysis software packages such as SAS or R to perform tasks such as determining relationships between variables or identifying anomalies in data sets.
- Reviewing and analyzing data sets to identify patterns or trends that may have business implications.
- Developing data mining solutions by writing computer code or using tools such as Hive, Pig, or MapReduce.
Management and Strategy:
- Overseeing activities of the junior data engineering teams, ensuring proper execution of their duties and alignment with business vision and objectives.
- Provides a senior-level contribution to a team responsible for designing, deploying, and maintaining the business’s data platforms.
- Owns and extends the business’s data pipeline through the collection, storage, processing, and transformation of large data sets.
- Monitor the existing metrics, analyze data, and lead partnerships with other Data and Analytics teams to identify and implement the system and process improvements.
- Develop queries for ad hoc business projects, as well as ongoing reporting.
- Builds a metadata system where all available data is maintained and cataloged.
- Responsible for developing reliable data pipelines that translate raw data into powerful features and signals.
- Designing, architecting, implementing, and supporting key datasets that avail structured and timely access to actionable business insights.
- Developing ETL processes that convert data into formats through a team of data analysts and dashboard charts.
Requirements
- Bachelor’s degree in computer science, engineering, or related field
- 5+ years professional experience as a data engineer or similar role
- Proficient in SQL, Python, and Java
- Experience with big data processing tools, such as Hadoop, Spark, and MapReduce
- Strong Experience with Cloud Manage Service such as Amazon RDS , Amazon MSK , Kinesis
- Experience with cloud-based data storage and processing solutions, such as Amazon S3 and Amazon EMR
- Strong problem-solving and critical thinking skills
- Data analysis: Data analysis is the ability to interpret data and find patterns or trends. Data engineers often use data analysis to determine how to best implement data security, data storage and data transfer. Data analysis can also help data engineers determine the best software to use for data analysis.
- Database management: Database management is the ability to create and maintain databases. This is a crucial skill for senior data engineers because they often work with large databases that contain important information. Senior data engineers with strong database management skills can create and maintain databases that are secure and efficient.
- Algorithms: Algorithms are the steps used to solve a problem. Data engineers use algorithms to solve technical problems and create software. Senior data engineers may need to create algorithms from scratch, so they need to understand how to do so.
- Machine learning: Machine learning is the ability to use data to predict future outcomes. This is a valuable skill for senior data engineers because it allows them to make more informed decisions about how to improve their company’s data systems.
- Communication: Data engineers often work with other engineers and other departments to complete projects. Effective communication is crucial to collaborating with others and ensuring everyone understands each other. Data engineers should also communicate with clients to explain technical processes and answer questions.
Preferred Skills and Qualifications
- Master’s degree in computer science, engineering, or related field
- 7+ years professional experience as a data engineer or similar role
- Experience with NoSQL databases, such as MongoDB, Cassandra, and HBase
- Experience with data visualization tools, such as Tableau and QlikView
- Experience with machine learning and predictive modeling
