Azure and Databricks Data Engineer

A well-established company that is determined to revolutionize the healthcare industry with their innovative technology and patient care systems, is looking for an Azure and Databricks Data Engineer. The engineer will be in charge of ensuring data quality, consistency, and reliability by performing data transformation and integration processes. The U.S.-based organization is providing excellent behavioral health and addiction services and solutions that have put them at the forefront of global healthcare treatment. This position requires at least 4-6+ hours of overlap with the CST time zone. 

Job Responsibilities:

• Utilizing the Azure ecosystem and Databricks platform, create, build, and maintain data engineering solutions
• Build scalable and effective data pipelines, perform data transformation and integration, and enable sophisticated analytics and machine learning workflows while collaborating closely with cross-functional teams that include data scientists, analysts, and software engineers
• Utilize your knowledge of Databricks and Azure services to support business intelligence efforts and deliver data-driven insights
• On the Azure platform with Databricks, help to design, create, and deploy end-to-end data engineering solutions
• With the help of stakeholders, collect and interpret business requirements into technical data engineering solutions
• Execute data ingestion operations, which involve gathering data from diverse sources, processing it to suit business requirements, and feeding it into data lakes or data warehouses
• Utilizing Azure Data Factory, Databricks, and other pertinent tools, create and maintain scalable data pipelines and workflows
• For effective data storage and retrieval, create and improve data models and schemas
• Implement security measures, and make sure all legal and regulatory standards are met regarding data protection

Job Requirements:

• Bachelor’s/Master’s degree in Engineering, Computer Science (or equivalent experience)
• At least 6-8+ years of relevant experience as a data engineer
• Prolific experience working with Azure and Databricks
• In-depth knowledge of the principles, procedures, and best practices of data engineering
• Demonstrable experience leveraging Azure Data Factory, Databricks, and other pertinent technologies to develop and implement data pipelines
• Proficient knowledge of programming languages like Python, SQL, and Scala 
• Thorough knowledge of data processing and storage technologies like Apache Spark, Azure Data Lake Storage, and Azure SQL Data Warehouse
• Familiarity with ETL/ELT procedures, schema design, and data modeling
• Proven problem-solving and troubleshooting abilities, including the ability to recognize and fix data-related problems
• Outstanding teamwork and communication abilities to work well with cross-functional teams.
• Data engineering-related Azure certificates or Databricks certifications are nice to have
• Azure credentials in Machine Learning or DevOps are desirable
• Prior remote project management experience is crucial
• Fluent in spoken and written English communication