Senior Data Engineer

A U.S.-based company that is on a mission to establish itself as a world-class distributor of organic produce to loyal customers, is looking for a Senior Data Engineer. The engineer will be performing data source gap analysis and developing data source/target catalogs and mappings. The company is providing an online subscription-based marketplace for natural and organic food products at a reasonable price. The company has managed to secure more than $240mn in funding so far. This position requires 6-8+ hours of overlap with the PST time zone. 

Job Responsibilities:

• Work with diverse functional teams to coordinate and deliver cohesive data engineering solutions across many projects and endeavors
• Hands-on experience working on structured/semi-structured
• Help the team to execute the Data Engineering and Analytics initiatives and turn them into products that will provide great value to our members
• Complex data processing and streaming frameworks
• Work with other technical teams to produce data solutions that satisfy business and technical objectives by being hands-on and taking ownership of the whole cycle of data services, from data ingestion to data processing to ETL to data distribution for reporting
• Establish the technical specifications and specifics of the implementation for the underlying data lake, data warehouse, and data marts
• Perform data source gap analysis and create data source/target to identify, troubleshoot, and fix production data integrity and performance issues
• Collaborate with all areas of data management as lead to ensure that patterns, decisions, and tooling are implemented in accordance with enterprise standards
• To establish an enterprise picture of the company's future data demands, thoroughly grasp cross-system integration, interactions, and relationships
• Recommend/Ensure technical functionality (e.g. scalability, security, performance, data recovery, reliability, etc.) for Data Engineering
• Facilitate workshops to define requirements and develop data solution designs
• Apply enterprise and solution architecture decisions to data architecture frameworks and data models
• Maintain a repository efficiently
• Design, coordinate, and execute pilots, prototypes, and proofs of concept to provide validation on specific scenarios and provide an implementation roadmap
• Work together to predict and create data architecture that meets business demands for the collection, aggregation, and interaction with multiple data sources with IT teams software suppliers and business owners

Job Requirements:

• Bachelor’s/Master’s degree in Engineering, Computer Science (or equivalent experience)
• At least 5+ years of relevant experience as a data engineer
• Extensive experience working with Python, Scala, Airflow, and SQL
• Prolific experience with Data warehousing/modeling
• Nice to have some experience with Git, AWS Redshift, and Microservices
• Familiarity with DBT (Data Build Tool) is desirable
• Ability to work cross-functionally with other engineering teams
• Expertise with RDBMS and Data Warehousing (Strong SQL) with Redshift, Snowflake, or similar
• In-depth knowledge and experience with data and information architecture patterns and implementation approaches for Operational Data Stores, Data Warehouses, Data Marts, and Data Lakes
• Proficiency in logical/physical data architecture, design, and development
• Experience in Data lake / Big data analytics platform implementation either cloud-based or on-premise; AWS preferred
• Experience working with high volumes of data; experience in design, implementation, and support of highly distributed data applications
• Experience with Development Tools for CI/CD, Unit and Integration testing,
• Automation and Orchestration E.g. GitHub, Jenkins, Concourse, Airflow, Terraform
• Experience with writing Kafka producers and consumers or experience with AWS Kinesis
• Hands-on experience developing a distributed data processing platform with Big Data technologies like Hadoop, Spark, etc
• A knack for independence (hands-on) as well as team work
• Excellent analytical and problem-solving skills, often in light of ill-defined issues or conflicting information.
• Experience with streaming data ingestion, machine learning, and Apache Spark a plus
• Adept in the ability to elicit, gather, and manage requirements in an Agile delivery environment
• Excellent communication and presentation skills (verbal, written, presentation) across all levels of the organization
• Ability to translate ambiguous concepts into tangible ideas
• Excellent English communication skills