The Compass Digital Labs Data Engineer interview guide, interview questions, salary data, and interview experiences.

Compass Digital Labs Data Engineer Interview Guide

Getting ready for an Data Engineer interview at Compass Digital Labs? The Compass Digital Labs Data Engineer interview span across 10 to 12 different question topics. In preparing for the interview:
<ul>
<li>Know what skills are necessary for Compass Digital Labs Data Engineer roles.</li>
<li>Gain insights into the Data Engineer interview process at Compass Digital Labs.</li>
<li>Practice real Compass Digital Labs Data Engineer interview questions.</li>
</ul>
Interview Query regularly analyzes interview experience data, and we've used that data to produce this guide, with sample interview questions and an overview of the Compass Digital Labs Data Engineer interview.


Getting ready for an Data Engineer interview at Compass Digital Labs? The Compass Digital Labs Data Engineer interview span across 10 to 12 different question topics. In preparing for the interview:

- Know what skills are necessary for Compass Digital Labs Data Engineer roles.
- Gain insights into the Data Engineer interview process at Compass Digital Labs.
- Practice real Compass Digital Labs Data Engineer interview questions.

Interview Query regularly analyzes interview experience data, and we've used that data to produce this guide, with sample interview questions and an overview of the Compass Digital Labs Data Engineer interview.
        

Overview

What Compass Digital Labs Looks for in a Data Engineer

We've gathered this data from parsing thousands of interview experiences sourced from members.

We've gathered this data from parsing thousands of interview experiences sourced from members.

Compass Digital Labs Data Engineer Interview Process

Interview Tips

You are tasked with keeping track of a customer’s addresses in your database.

However, customer’s addresses change whenever they move. We want to keep track of their moving history as well as the person that moves in afterwards.

How would you create or modify a schema to keep track of these address changes?

Create a schema to keep track of customer address changes

Address Schema

Data Modeling

You work as a data engineer for a city-wide bicycle rental company. The company wants to predict future bicycle rental volumes using historical data collected from multiple sources, such as station logs, weather files, and event schedules. You are given access to raw source files that are updated daily.

How would you design an end-to-end data pipeline to prepare and serve data for downstream predictive modeling of bicycle rental demand?

Design an end-to-end data pipeline to process and serve data for predicting bicycle rental volumes.

Bicycle Rental Data Pipeline

Batch & Stream Processing

You’re tasked with designing a database system for Swipe Inc., a company that manages payments for software developers. Swipe manages a set of APIs so developers can abstract payment processes for their web service and transfer control over to Swipe’s APIs to manage payment security and finance handling (contacting credit card providers, etc.)

Your database system should store a collection of API keys with their attributes, as shown in the example below:

<pre tabindex="0" class="chroma"><code>{
 &#34;API_key&#34;:{
 &#34;Transaction_list&#34;:{
 &#34;User_ID&#34;:{
 &#34;Name&#34;:{
 &#34;f_name&#34;: 
 &#34;l_name&#34;: 
 }
 &#34;Card_Details&#34;:{
 &#34;Bank&#34;:
 &#34;Card_No&#34;:
 &#34;Expiry&#34;:
 &#34;CV&#34;:
 } 
 }
 }
 &#34;Curr_balance&#34;:
 }
}
</code></pre>

What are the functional and non-functional requirements for this database?

Determine the requirements for designing a database system to store payment APIs

Swipe Payment API

Compass Digital Labs Data Engineer Interview Questions

<h3>Job Description</h3><div>Job Description<div> We are Compass Digital—the digital and technology arm of Compass Group North America. We build and scale digital products and technologies to exceed the expectations of our guests and clients. Backed by research, we create remarkable user experiences with increased choice and convenience. We offer end-to-end solutions, simultaneously boosting sales and guest satisfaction.  Compass Digital was developed from within Compass Group, focusing on driving transformation and innovation in the hospitality spaces across Business & Industry, Education, and Healthcare. We power digital in the hospitality experience, ensuring that each touchpoint is optimized for guest satisfaction while leveraging data to achieve additional outcomes.  We are fully integrated into Compass Group, proudly serving its’ various clients through scalable, cutting-edge technologies and solutions.<div><div><div>Job Summary</div><div>We're looking for a hands-on, collaborative Data Engineer to join our Data Services team. In this role you'll have the opportunity to build and troubleshoot ingestion pipelines for multiple sources of data. You will also work closely with our business intelligence and reporting solutions team to help develop valuable data models and visualizations for the business. We’re currently leveraging AWS cloud and using a range of AWS services like S3, DynamoDB, Athena, Redshift, SQS, Lambda and Glue. However, we are migrating to Snowflake as our primary data warehouse platform for new projects and developing a new data pipeline architecture</div></div><div><div>Job Responsibilities </div><div>· Work with data analysts to build pipelines for required projects and requests· Define, build, test, and implement scalable data pipelines using python and SQL· Monitor pipeline performance and efficiency· Troubleshoot data related issues including missing data, poor data quality, etc.· Collaborate with analytics and reporting teams to develop data models that feed business intelligence tools· Maintain code via CI/CD process as defined in our Azure DevOps platform</div></div><div><div>Job Qualifications</div><div>· 3-5 year experience in a similar role· Good Communication and open to learning new skills· Bachelor’s degree or equivalent in Computer Science, Computer Engineering, Information Systems or similar· Programming skills with Python, Spark (bonus)· Experience with Airflow or similar system.· Experience with public cloud provider (AWS prefer)· Knowledge of SQL, relational database concepts and SQL scripting· Experience with Docker, Kubernetes, Spark, Snowflake, CI/CD a bonus</div></div></div>Apply to Compass Group today!Click here to Learn More about the Compass Story Compass Group is an equal opportunity employer.  At Compass, we are committed to treating all Applicants and Associates fairly based on their abilities, achievements, and experience without regard to race, national origin, sex, age, disability, veteran status, sexual orientation, gender identity, or any other classification protected by law.Qualified candidates must be able to perform the essential functions of this position satisfactorily with or without a reasonable accommodation. Disclaimer: this job post is not necessarily an exhaustive list of all essential responsibilities, skills, tasks, or requirements associated with this position. While this is intended to be an accurate reflection of the position posted, the Company reserves the right to modify or change the essential functions of the job based on business necessity. We will consider for employment all qualified applicants, including those with a criminal history (including relevant driving history), in a manner consistent with all applicable federal, state, and local laws, including the City of Los Angeles’ Fair Chance Initiative for Hiring Ordinance, the San Francisco Fair Chance Ordinance, and the New York Fair Chance Act.  Compass Digital maintains a drug-free workplace. Applications are accepted on an ongoing basis. Associates at Corporate are offered many fantastic benefits. <ul><li>Medical</li><li>Dental</li><li>Vision</li><li>Life Insurance/ AD</li><li>Disability Insurance</li><li>Retirement Plan</li><li>Paid Time Off</li><li>Holiday Time Off (varies by site/state)</li><li>Associate Shopping Program</li><li>Health and Wellness Programs</li><li>Discount Marketplace</li><li>Identity Theft Protection</li><li>Pet Insurance</li><li>Commuter Benefits</li><li>Employee Assistance Program</li><li>Flexible Spending Accounts (FSAs)</li><li>Paid Parental Leave</li><li>Personal Leave</li></ul> Associates may also be eligible for paid and/or unpaid time off benefits in accordance with applicable federal, state, and local laws. For positions in Washington State, Maryland, or to be performed Remotely, click here for paid time off benefits information.Req ID:  1448722Compass Digital Bankston B Williams </div></div>

Job Title: Data Engineer Location: Irvine, CA (Onsite for the first 90 days, then Hybrid)Duration: Direct-Hire Opportunity Position Summary We are looking for a Data Engineer who is hands-on, collaborative, and experienced with Microsoft SQL Server, Snowflake, AWS RDS, and MySQL. The ideal candidate has a strong background in data warehousing, data lakes, ETL pipelines, and business intelligence tools. This role plays a key part in executing data strategy - driving optimization, reliability, and scalable BI capabilities across the organization. It's an excellent opportunity for a data professional who wants to influence architectural direction, contribute technical expertise, and grow within a data-driven company focused on innovation. Key Responsibilities<ul><li>Design, develop, and maintain SQL Server and Snowflake data warehouses and data lakes, focusing on performance, governance, and security.</li><li>Manage and optimize database solutions within Snowflake, SQL Server, MySQL, and AWS RDS.</li><li>Build and enhance ETL pipelines using tools such as Snowpipe, DBT, Boomi, SSIS, and Azure Data Factory.</li><li>Utilize data tools such as SSMS, Profiler, Query Store, and Redgate for performance tuning and troubleshooting.</li><li>Perform database administration tasks, including backup, restore, and monitoring.</li><li>Collaborate with Business Intelligence Developers and Business Analysts on enterprise data projects.</li><li>Ensure database integrity, compliance, and adherence to best practices in data security.</li><li>Configure and manage data integration and BI tools such as Power BI, Tableau, Power Automate, and scripting languages (Python, R).</li></ul> Qualifications<ul><li>Proficiency with Microsoft SQL Server, including advanced T-SQL development and optimization.</li><li>7+ years working as a SQL Server Developer/Administrator, with experience in relational and object-oriented databases.</li><li>2+ years of experience with Snowflake data warehouse and data lake solutions.</li><li>Experience developing pipelines and reporting solutions using Power BI, SSRS, SSIS, Azure Data Factory, or DBT.</li><li>Scripting and automation experience using Python, PowerShell, or R.</li><li>Familiarity with data integration and analytics tools such as Boomi, Redshift, or Databricks (a plus).</li><li>Excellent communication, problem-solving, and organizational skills.</li><li>Education: Bachelor's or Master's degree in Computer Science, Information Systems, Data Science, or a related field.</li></ul> Technical Skills<ul><li>SQL Server / Snowflake / MySQL / AWS RDS</li><li>ETL Development (Snowpipe, SSIS, Azure Data Factory, DBT)</li><li>BI Tools (Power BI, Tableau)</li><li>Python, R, PowerShell</li><li>Data Governance & Security Best Practices</li></ul> Determining compensation for this role (and others) at Vaco/Highspring depends upon a wide array of factors including but not limited to the individual's skill sets, experience and training, licensure and certifications, office location and other geographic considerations, as well as other business and organizational needs. With that said, as required by local law in geographies that require salary range disclosure, Vaco/Highspring notes the salary range for the role is noted in this job posting. The individual may also be eligible for discretionary bonuses, and can participate in medical, dental, and vision benefits as well as the company's 401(k) retirement plan. Additional disclaimer: Unless otherwise noted in the job description, the position Vaco/Highspring is filing for is occupied. Please note, however, that Vaco/Highspring is regularly asked to provide talent to other organizations. By submitting to this position, you are agreeing to be included in our talent pool for future hiring for similarly qualified positions. Submissions to this position are subject to the use of AI to perform preliminary candidate screenings, focused on ensuring minimum job requirements noted in the position are satisfied. Further assessment of candidates beyond this initial phase within Vaco/Highspring will be otherwise assessed by recruiters and hiring managers. Vaco/Highspring does not have knowledge of the tools used by its clients in making final hiring decisions and cannot opine on their use of AI products.

Primary Responsibilities:Design and implement automated test suites for AI/ML workflows that enables Optum to derive information from patient data in a scalable, reliable, and cost-effective manner using AIAnalyze clinical data and determine the best designs on how to verify the correct processing of this dataPerform functional, regression, integration, and system testing to validate software qualityIdentify, create, and track reproducible defects using bug-tracking tools, collaborating with developers for resolutionConduct performance and load testing to identify bottlenecks and measure throughput Integrate automated tests to our AWS and GCP accounts to achieve continuous integration (CI/CD)As an advocate for product quality, ensure that team members adopt agile SCRUM methodologies, perform unit tests, code reviewsReview and contribute to engineering specifications to ensure that each feature/user story can be tested in an automated fashionWrite integration test plans with end-end test automation as a goalMentoring a couple of mode Junior engineers who may write test casesDesign, develop, and deploy AI-powered solutions using no-code, low-code, and advanced platforms, translating business needs into scalable applications that enhance products, workflows and decision-making You’ll be rewarded and recognized for your performance in an environment that will challenge you and give you clear direction on what it takes to succeed in your role as well as provide development for other roles you may be interested in. Required Qualifications:Master’s degree in computer science, Engineering, or a related technical field4+ years of work experience in AI/ML engineering with solid proficiency in Python programming2+ years of experience as an SDET using test automation tools like Selenium, Cypress or similar toolsExperience in at least one project using API testing tools like Selenium, Pytest, Postman or similarDelivered at least one product hosted on a cloud platform (AWS, GCP, or Azure) Preferred Qualifications: AWS experience such as RDS, ALB, Redis Cache, Secret Manager, EC2, IAMExperience in deploying generative AI systems (LLMs, Prompts, Agentic systems)Experience with Rally Experience integrating with healthcare data systems and working with clinical data formats Solid communication and collaboration skills, with the ability to thrive in fast-paced and ambiguous environmentsSolid analytical skills and ability to debug issues in complex systemsExperience with CI/CD pipelines and infrastructure-as-code tools (e.g., Terraform, CloudFormation, GitHub actions, Azure DevOps) *All employees working remotely will be required to adhere to UnitedHealth Group’s Telecommuter Policy. Pay is based on several factors including but not limited to local labor markets, education, work experience, certifications, etc. In addition to your salary, we offer benefits such as, a comprehensive benefits package, incentive and recognition programs, equity stock purchase and 401k contribution (all benefits are subject to eligibility requirements). No matter where or when you begin a career with us, you’ll find a far-reaching choice of benefits and incentives. The salary for this role will range from $89,900 to $160,600 annually based on full-time employment. We comply with all minimum wage laws as applicable. At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone–of every race, gender, sexuality, age, location and income–deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes — an enterprise priority reflected in our mission. UnitedHealth Group is an Equal Employment Opportunity employer under applicable law and qualified applicants will receive consideration for employment without regard to race, national origin, religion, age, color, sex, sexual orientation, gender identity, disability, or protected veteran status, or any other characteristic protected by local, state, or federal laws, rules, or regulations. UnitedHealth Group is a drug - free workplace. Candidates are required to pass a drug test before beginning employment.

Job Title: Python Data Engineer Location: Houston, TX (5 days onsite)Experience & Skills<ul><li>5+ years in Data Engineering with strong SQL and NoSQL database skills:<ul><li>Databases: Oracle, SQL Server, Postgres, DB2, Elasticsearch, MongoDB</li></ul></li><li>Advanced Python development and FastAPI microservices experience</li><li>Application development experience implementing business logic via SQL stored procedures and NoSQL utilities</li><li>Experience designing scalable and performant processes:<ul><li>Must provide metrics: transactions/day, largest DB table size, concurrent users, API response times</li></ul></li><li>Real-time interactive applications with UI-to-database communication:<ul><li>Must explain protocols and data formats used (e.g., JSON, REST, WebSockets)</li></ul></li><li>Experience using LLM models, coding agents, and testing agents:<ul><li>Provide specific examples of problem-solving</li></ul></li><li>Ability to handle support and development simultaneously:<ul><li>Detail daily split between support and development, ticketing system usage, or direct user interaction</li></ul></li><li>Bachelor's degree in Computer Science or relevant major</li><li>Strong analytic skills, AI tool usage, multitasking, self-management, and direct collaboration with business users </li></ul>Not a Good Fit <ul><li>Experience limited to ETL / backend processes / data transfer between databases</li><li>Experience only on cloud platforms (Azure, AWS, GCP) without SQL/NoSQL + Python expertise </li></ul>Dexian stands at the forefront of Talent + Technology solutions with a presence spanning more than 70 locations worldwide and a team exceeding 10,000 professionals. As one of the largest technology and professional staffing companies and one of the largest minority-owned staffing companies in the United States, Dexian combines over 30 years of industry expertise with cutting-edge technologies to deliver comprehensive global services and support.     Dexian connects the right talent and the right technology with the right organizations to deliver trajectory-changing results that help everyone achieve their ambitions and goals. To learn more, please visit <a>https://dexian.com/</a>. Dexian is an Equal Opportunity Employer that recruits and hires qualified candidates without regard to race, religion, sex, sexual orientation, gender identity, age, national origin, ancestry, citizenship, disability, or veteran status.

???? We’re Hiring: Data Engineer (Azure) – Porto, Portugal (Hybrid)Are you passionate about building scalable data solutions and working with cutting-edge cloud technologies? Join a leading name in Automotive Technology and help shape the future of data-driven innovation. Job Title: Data Engineer (Azure)Sector: Automotive TechnologyLocation: Porto, Portugal (Hybrid)Languages: English (Fluent) & Portuguese-Nice to have About the RoleWe are looking for an experienced Data Engineer to join our team in Porto. This role involves designing and maintaining scalable data solutions in a dynamic automotive technology environment. Key Responsibilities<ul><li>Design, build, and maintain data ingestion, transformation, and delivery pipelines using Azure.</li><li>Work with structured and unstructured data from multiple sources.</li><li>Ensure efficiency, scalability, and reliability of data solutions.</li><li>Collaborate with analysts and data scientists to make data accessible and analysis-ready.</li><li>Implement best practices for data quality, lineage, and security.</li><li>Develop monitoring tools and alerts for seamless pipeline operations.</li><li>Maintain clear documentation of data architectures and technical decisions.</li></ul> Key Skills & Experience<ul><li>Proven experience as a Data Engineer in cloud environments, especially Microsoft Azure.</li><li>Hands-on experience with:</li><li>Azure Data Factory, Synapse Analytics, Data Lake Storage Gen2</li><li>Azure SQL Database, Databricks, Azure DevOps (CI/CD)</li><li>Proficient in SQL and Python; knowledge of Scala or Apache Spark is a plus.</li><li>Strong analytical skills and attention to detail.</li><li>Comfortable working in agile, cross-functional teams.</li></ul> ???? Interested?Connect with me on LinkedIn or apply directly via email: deepshikha.jain3@hays.com

Work Location- Gurugram, Pune, Bangalore Key Role and Responsibilities: <ul><li>Design, develop, test, and deploy data integration processes (batch or real-time) using tools such as Microsoft SSIS, Azure Data Factory, Databricks, Matillion, Airflow, Sqoop, etc.</li><li>•Create functional & technical documentation e.g. ETL architecture documentation, unit testing plans and results, data integration specifications, data testing plans, etc.</li><li>Provide a consultative approach with business users, asking questions to understand the business need and deriving the data flow, conceptual, logical, and physical data models based on those needs. Perform data analysis to validate data models and to confirm ability to meet business needs.</li><li>May serve as project or DI lead, overseeing multiple consultants from various competencies</li><li>Stays current with emerging and changing technologies to best recommend and implement beneficial technologies and approaches for Data Integration</li><li>Ensures proper execution/creation of methodology, training, templates, resource plans and engagement review processes</li><li>Coach team members to ensure understanding on projects and tasks, providing effective feedback (critical and positive) and promoting growth opportunities when appropriate.</li><li>Coordinate and consult with the project manager, client business staff, client technical staff and project developers in data architecture best practices and anything else that is data related at the project or business unit levels</li><li>Architect, design, develop and set direction for enterprise self-service analytic solutions, business intelligence reports, visualisations and best practice standards. Toolsets include but not limited to: SQL Server Analysis and Reporting Services, Microsoft Power BI, Tableau and Qlik.</li><li>Work with report team to identify, design and implement a reporting user experience that is consistent and intuitive across environments, across report methods, defines security and meets usability and scalability best practice.</li></ul>

Job Title / Primary Skill: Data EngineerYears of Experience: 2.5 to 5 yearsJob Location: Pune (Hybrid)Must Have Skills: Data Engineer, Databricks, Python/Pyspark, SQL, AWS/AzureEducational Qualification: BE/BTech/ MTech/ MCA (Minimum bachelor’s degree) Job OverviewAs a Data Engineering Consultant, you will be instrumental in designing, developing, and optimizing robust data engineering solutions that support critical business analytics and reporting. You will provide high-quality, timely development and on-time input for service delivery, managed services, or implementation-oriented client projects. Assignments will range from foundational analysis and problem-solving to contributing to the architecture of complex data solutions and potentially leading small technical teams. Your contributions will directly impact client satisfaction objectives. You will also be responsible for developing technical modules and low-level designs while monitoring the technical quality standards of onshore/offshore development teams. Essential Functions<ul><li>Designing and Building Data Warehouse Solutions: Design and implement robust ETL/ELT pipelines to process structured and unstructured data. Support the end-to-end design and build of new data warehousing solutions, including understanding intricate business data needs, translating them into technical specifications, and planning for efficient data integration and delivery.</li><li>Requirements Gathering and Data Modelling: Conduct in-depth analysis of source systems and business requirements to generate detailed data warehousing specifications, including conceptual, logical, and physical data models. Create comprehensive technical documentation, including design specifications, data flow diagrams, and mapping documents.</li><li>Stakeholder Collaboration: Manage internal and external cross-functional stakeholders, including business users, data architects, ETL developers, and BI specialists, to ensure successful data warehouse development and adoption.</li><li>Strategic Data Architecture: Develop scalable data architectures using technologies like Spark, Hadoop, and cloud-native tools (AWS, Azure). Leverage deep technical expertise and business acumen to identify strategic alternatives and work with project lead to recommend optimal data warehousing architectures that align with client business goals.</li><li>Knowledge Development: Continuously develop a broad knowledge of data warehousing methodologies, industry best practices, and emerging technologies (e.g., cloud data platforms, data lakes) through project delivery and formal/informal learning opportunities.</li><li>Independent and Team Work: Work effectively both independently and as a vital part of a technical team, contributing to all phases of the data warehousing lifecycle.</li><li>Client Engagement: Attend, support, and present technical solutions and progress updates at client meetings.</li><li>Mentorship and Knowledge Sharing: Contribute to the learning, development, and mentoring of junior team members.</li></ul> Qualifications: Educational Background:<ul><li>Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related quantitative field.</li><li>Master's Degree in a related field preferred.</li></ul> Technical Expertise:<ul><li>Proven experience in designing, developing, and implementing enterprise data warehouses using various database technologies (e.g., SQL Server, Oracle, PostgreSQL, Snowflake, Redshift, Azure Synapse).</li><li>Strong proficiency in SQL and Python</li><li>Experience with big data tools (e.g., Spark, Hadoop, Hive).</li></ul> Preferred Qualifications:<ul><li>Prior work experience in pharmaceutical domain</li><li>Hands-on experience in project management tools (Jira, Asana, Smartsheet, etc.)</li><li>Team management - Experience of leading small project teams (2-3 members)</li><li>Knowledge of Generative AI concepts and tools</li><li>Strong hands-on experience in either of the cloud platforms - AWS, Azure.</li><li>Understanding of data governance, data quality, and master data management principles.</li><li>Experience working in Agile / Scrum environments.</li></ul> Professional Attributes:<ul><li>Works willingly and effectively with others in and across the organization to accomplish team goals.</li><li>Knowledge and understanding of the fundamental processes of business, their interaction, and the impact of external/internal influences on decision-making, growth, and decline, specifically as they relate to data strategy.</li><li>Knowledge of consulting methods, tools, and techniques related to data warehousing and business intelligence.</li><li>Knowledge of current events and developments within the data management industry and major competitors.</li><li>Effective time management and project organization skills, with the ability to manage multiple priorities.</li></ul>

Data Engineer | Brussels (Hybrid) | Up to €6500 Join our growing team as we expand into international markets and build the data foundation powering our online platform. What we need <ul><li>Design and optimise SQL queries, data models, and transformations to support analytics and operations.</li><li>Build internal dashboards and tools in React (Recharts.js / D3) to empower marketing, support, and product teams.</li><li>Develop and maintain data pipelines, CDC workflows, and high-scale datasets.</li><li>Integrate and support BI tools as our data stack evolves.</li><li>Work cross-functionally to understand business needs: acquisition, retention, segmentation and turn them into actionable metrics and insights.</li><li>3+ years experience </li></ul> What You Bring <ul><li>Deep expertise in SQL and relational database design.</li><li>Experience with React and at least one backend/full-stack language.</li><li>Understanding of data warehousing and orchestration tools.</li><li>Ability to work with large datasets in a high-traffic environment.</li><li>Strong collaboration skills and interest in the online gaming/entertainment domain.</li></ul> Nice to Have <ul><li>Experience with BI platforms.</li><li>Knowledge of marketing analytics, CRM, funnel analysis, or customer segmentation.</li><li>Ability to define KPIs and propose impactful dashboards.</li></ul> Feel free to apply here or message me directly.

Job Overview: We are looking for a skilled and motivated Lead Data Engineer with strong experience in Python programming and Google Cloud Platform (GCP) to join our data engineering team. The ideal candidate will be responsible for requirements gathering, designing, architecting the solution, developing, and maintaining robust and scalable ETL (Extract, Transform, Load) & ELT data pipelines. The role involves working with customers directly, gathering requirements, discovery phase, designing, architecting the solution, using various GCP services, implementing data transformations, data ingestion, data quality, and consistency across systems, and post post-delivery support. Experience Level: 10 to 12 years of relevant IT experienceKey Responsibilities:<ul><li>Design, develop, test, and maintain scalable ETL data pipelines using Python.</li><li>Architect the enterprise solutions with various technologies like Kafka, multi-cloud services, auto-scaling using GKE, Load balancers, APIGEE proxy API management, DBT, using LLMs as needed in the solution, redaction of sensitive information, DLP (Data Loss Prevention) etc.</li><li>Work extensively on Google Cloud Platform (GCP) services such as:</li><li>Dataflow for real-time and batch data processing</li><li>Cloud Functions for lightweight serverless compute</li><li>BigQuery for data warehousing and analytics</li><li>Cloud Composer for orchestration of data workflows (on Apache Airflow)</li><li>Google Cloud Storage (GCS) for managing data at scale</li><li>IAM for access control and security</li><li>Cloud Run for containerized applications</li><li>Should have experience in the following areas : </li><li>API framework: Python FastAPI</li><li>Processing engine: Apache Spark</li><li>Messaging and streaming data processing: Kafka</li><li>Storage: MongoDB, Redis/Bigtable</li><li>Orchestration: Airflow</li><li>Experience in deployments in GKE, Cloud Run.</li><li>Perform data ingestion from various sources and apply transformation and cleansing logic to ensure high-quality data delivery.</li><li>Implement and enforce data quality checks, validation rules, and monitoring.</li><li>Collaborate with data scientists, analysts, and other engineering teams to understand data needs and deliver efficient data solutions.</li><li>Manage version control using GitHub and participate in CI/CD pipeline deployments for data projects.</li><li>Write complex SQL queries for data extraction and validation from relational databases such as SQL Server, Oracle, or PostgreSQL.</li><li>Document pipeline designs, data flow diagrams, and operational support procedures.</li></ul>Required Skills:<ul><li>10 to 12 years of hands-on experience in Python for backend or data engineering projects.</li><li>Strong understanding and working experience with GCP cloud services (especially Dataflow, BigQuery, Cloud Functions, Cloud Composer, etc.).</li><li>Solid understanding of data pipeline architecture, data integration, and transformation techniques.</li><li>Experience in working with version control systems like GitHub and knowledge of CI/CD practices.</li><li>Experience in Apache Spark, Kafka, Redis, Fast APIs, Airflow, GCP Composer DAGs.</li><li>Strong experience in SQL with at least one enterprise database (SQL Server, Oracle, PostgreSQL, etc.).</li><li>Experience in data migrations from on-premise data sources to Cloud platforms.</li></ul>Good to Have (Optional Skills):<ul><li>Experience working with the Snowflake cloud data platform.</li><li>Hands-on knowledge of Databricks for big data processing and analytics.</li><li>Familiarity with Azure Data Factory (ADF) and other Azure data engineering tools.</li></ul>Additional Details:<ul><li>Excellent problem-solving and analytical skills.</li><li>Strong communication skills and ability to collaborate in a team environment.</li></ul>Education:<ul><li> Bachelor's degree in Computer Science, a related field, or equivalent experience.</li></ul>

Komatsu is one of the largest worldwide manufacturers of construction, mining and utility equipment, and also delivers forest machines and logistics. They have manufacturing operations across the globe, including in Japan, Asia, America and Europe. Komatsu has been present in Europe since 1967, where they have several production plants. The headquarters of Komatsu Europe is located in Vilvoorde. KEISA:Komatsu Europe International, located in Vilvoorde, is the European Headquarter of the Komatsu Group, which is based in Tokyo, Japan. After first being located in Antwerp and Haren (Brussels), KEISA was set up in Vilvoorde in 1973. They import and distribute Komatsu construction, mining and utility equipment to European and North African countries and supervise European legal and financial operations. KEPO:Komatsu Europe Parts Operation is the Parts Distribution Center for Europe, the Middle East and Africa. Stocks of Komatsu Genuine Parts and consumables are maintained centrally at the KEPO warehouse, in Vilvoorde. A sophisticated central parts system is directly linked to Komatsu Sales and Service Organizations across Europe, Africa and the Middle East.Currently there are almost 300 people working at the Komatsu Europe Headquarters and Parts distribution center in Vilvoorde: technical experts, sales profiles, logistic and administrative staff, etc. RoleThe Data Engineer role exists to ensure reliable, secure, and business-ready access to data across Komatsu’s ICT landscape. Komatsu’s business decisions – from sales and finance to production and spare parts logistics – depend on having accurate, consistent, and timely data.Komatsu is undergoing one of its most significant ICT transformations in 25 years: migrating its on-premise data warehouse to a modern, cloud-based Snowflake platform. This transformation will become the strategic backbone for all BI, data analysis, and future AI initiatives within the global group. Without a strong Data Engineer to build and maintain the pipelines, clean and model the data, and ensure security and performance, the migration risks delays and inefficiencies that could directly impact business performance across Europe and beyond.The impact of this position is therefore immense: the Data Engineer will turn raw, siloed, and inconsistent data into trusted, actionable insights that drive Komatsu’s growth and competitiveness.Tasks and responsibilitiesAs a Data Engineer at Komatsu, your primary responsibility is to make sure that the business has access to reliable, secure, and business-ready data at all times. This is far more than a technical challenge: it is about enabling every department – from Sales and Marketing to Finance, Production, and Supply Chain – to make faster, smarter, and more confident decisions.You will take ownership of the full data lifecycle: from collecting raw, complex, and often fragmented data from operational ICT systems, through cleaning, transforming, and modeling it, to ensuring it flows into the central data warehouse (Snowflake) in a way that is performant, secure, and aligned with business requirements. This means you will be the “engineer behind the scenes” who guarantees that dashboards, reports, and advanced analytics are based on consistent, trustworthy, and well-structured data.While your focus will be data engineering (80%), you will also support colleagues in resolving advanced data questions and ensuring that BI tools (such as Qlik Sense and Power BI) can rely on high-quality datasets (20%). Projects<ul><li>Modern data warehouse project (Snowflake migration)</li></ul>The most critical and exciting aspect of this role is the opportunity to work on one of the most important ICT projects in Komatsu’s European history: the migration from an on-premise Microsoft SQL Server data warehouse to a modern, cloud-based Snowflake platform hosted on Azure. The current on-premise data warehouse has reached its limits in terms of scalability, flexibility, and performance. Snowflake has been chosen as the strategic backbone for BI, analytics, and AI. The migration involves moving around 20 different operational systems into Snowflake, with only ~20% completed so far. The Data Engineer will play a leading role in the remaining 80% of the migration, ensuring the delivery of clean, secure, and business-ready datasets.<ul><li>ERP migration (SAP)</li></ul>This project is tightly coupled with Komatsu’s global ERP transformation (migration to SAP). The upcoming transformation will be the largest ERP modernization in Komatsu in over 25 years. The Data Engineer will be involved in ensuring that all ERP-related data is integrated, modeled, and made available for analysis and reporting. Snowflake will provide the consolidated, validated, and historical data that the new ERP system requires, while also integrating real-time operational data. For the Data Engineer, this means being at the heart of a global transformation, directly impacting how Komatsu’s European operations (and eventually the global group) consume and leverage data.<ul><li>AI </li></ul>Komatsu is taking its first steps in AI and advanced analytics. AI models will only be as good as the data they are trained on – which means the Data Engineer’s role in preparing and delivering high-quality datasets is absolutely critical. You will be enabling projects that optimize operations, reduce costs, and create new business opportunities. ProfileBackground and experienceMust haves:<ul><li>Proven experience in data engineering, preferably within a data warehouse environment.</li><li>Experience with Snowflake or similar cloud-based data platforms is essential.</li></ul>Nice to haves:<ul><li>Knowledge of BI tools (Qlik Sense, Power BI, Tableau) though the role is not dashboard-focused.</li><li>Strong expertise in SQL, data modeling (multi-dimensional/star/snowflake schema), ETL/ELT processes, and database management.</li><li>Familiarity with Azure cloud architecture and DevOps practices is preferred.</li><li>Python knowledge is an advantage.</li><li>Experience in international and/or industrial environments is a plus.</li></ul>Competences<ul><li>Curious nature;</li><li>Drive and enthusiasm;</li><li>Eagerness to learn;</li><li>Hands-on approach;</li><li>Cultural sensitivity;</li><li>Highly social and collaborative;</li><li>Strong analytical and problem-solving skills;</li><li>Independence and ownership, combined with clear communication.</li></ul>Languages<ul><li>Proficiency in both English and Dutch is preferred. However, proficiency in English alone is also acceptable. </li></ul> Offer<ul><li>Be part of one of the most important ICT projects in Komatsu’s history;</li><li>Work on state-of-the-art cloud technologies (Snowflake, Azure, CI/CD) in a global organization;</li><li>Collaborate with international teams and contribute to projects with global reach;</li><li>The perfect fit for those who likes a challenge;</li><li>Steep and fast learning curve;</li><li>Engineer the foundation that enables Komatsu to innovate, grow, and stay competitive;</li><li>Enjoy a high degree of autonomy;</li><li>Access to extensive training and learning opportunities;</li><li>Exposure to AI and innovative projects in addition to core data engineering;</li><li>Join a forward-thinking company investing in digital transformation and technology innovation. The company emphasizes innovation in its product development, incorporating advanced technology to enhance efficiency, safety, and environmental sustainability;</li><li>Following your onboarding process, it will be possible to work 3 days per week from home;</li><li>Komatsu is renowned for producing high-quality and durable machinery and equipment used in various industries. Their products are known for their reliability and performance;</li><li>Collaborate with experts across departments and regions, backed by strong leadership and resources. With a global footprint, Komatsu operates in numerous countries, providing equipment and services worldwide, ensuring accessibility and support for customers globally.</li></ul>Amon is the exclusive recruitment partner for this position.If interested, please do not hesitate to contact Louise De Potter, ldp@amon.be

Question	Topic	Difficulty	Ask Chance
Address Schema	Data Modeling	Medium	Very High
Bicycle Rental Data Pipeline	Batch & Stream Processing	Medium	Very High
Swipe Payment API	Data Modeling	Easy	High