The Epic! Data Engineer interview guide, interview questions, salary data, and interview experiences.

Epic! Data Engineer Interview Guide

Getting ready for an Data Engineer interview at Epic!? The Epic! Data Engineer interview span across 10 to 12 different question topics. In preparing for the interview:
<ul>
<li>Know what skills are necessary for Epic! Data Engineer roles.</li>
<li>Gain insights into the Data Engineer interview process at Epic!.</li>
<li>Practice real Epic! Data Engineer interview questions.</li>
</ul>
Interview Query regularly analyzes interview experience data, and we've used that data to produce this guide, with sample interview questions and an overview of the Epic! Data Engineer interview.


Getting ready for an Data Engineer interview at Epic!? The Epic! Data Engineer interview span across 10 to 12 different question topics. In preparing for the interview:

- Know what skills are necessary for Epic! Data Engineer roles.
- Gain insights into the Data Engineer interview process at Epic!.
- Practice real Epic! Data Engineer interview questions.

Interview Query regularly analyzes interview experience data, and we've used that data to produce this guide, with sample interview questions and an overview of the Epic! Data Engineer interview.
        

Overview

We've gathered this data from parsing thousands of interview experiences sourced from members.

We've gathered this data from parsing thousands of interview experiences sourced from members.

Epic! Data Engineer Interview Process

You are tasked with keeping track of a customer’s addresses in your database.

However, customer’s addresses change whenever they move. We want to keep track of their moving history as well as the person that moves in afterwards.

How would you create or modify a schema to keep track of these address changes?

Create a schema to keep track of customer address changes

Address Schema

You’re tasked with designing a database system for Swipe Inc., a company that manages payments for software developers. Swipe manages a set of APIs so developers can abstract payment processes for their web service and transfer control over to Swipe’s APIs to manage payment security and finance handling (contacting credit card providers, etc.)

Your database system should store a collection of API keys with their attributes, as shown in the example below:

<pre tabindex="0" class="chroma"><code>{
 &#34;API_key&#34;:{
 &#34;Transaction_list&#34;:{
 &#34;User_ID&#34;:{
 &#34;Name&#34;:{
 &#34;f_name&#34;: 
 &#34;l_name&#34;: 
 }
 &#34;Card_Details&#34;:{
 &#34;Bank&#34;:
 &#34;Card_No&#34;:
 &#34;Expiry&#34;:
 &#34;CV&#34;:
 } 
 }
 }
 &#34;Curr_balance&#34;:
 }
}
</code></pre>

What are the functional and non-functional requirements for this database?

Determine the requirements for designing a database system to store payment APIs

Swipe Payment API

You are given a dictionary with a key-value of <code>{string: number}</code> where values in the dictionary could be duplicates. You are required to extract the unique values from the dictionary where the value occurred only once.

Return a list of values where they occur only once.

Note: You can return the values in any order.

Input:

<pre tabindex="0" class="chroma"><code>dictionary = {&#34;key1&#34;: 1, &#34;key2&#34;: 1, &#34;key3&#34;: 7, &#34;key4&#34;: 3, &#34;key5&#34;: 4, &#34;key6&#34;: 7}
</code></pre>

Output:

<pre tabindex="0" class="chroma"><code>find_unique_values(dictionary) -&gt; [3,4]

#Only 3 and 4 occurred once.
</code></pre>

Dictionary Unique Values

Epic! Data Engineer Interview Questions

Position Overview:   Reporting to the Head of Data Analytics & Strategy, the Data Engineer will design, implement, and maintain complex data storage, pipeline, and analytics systems that support and further develop the data infrastructure of the organization’s Quantitative and Fundamental investment groups. Responsibilities include building large data pipelines and developing APIs to retrieve data, all to innovate the next level of analytics systems that effectively integrate with and enhance current research applications   Responsibilities: •           Work with researchers and portfolio managers to analyze and improve the architecture of or proprietary and derived data sets •           Solve interesting data engineering problems in support of quantitative researches leveraging time-series data •           Implement cloud data platforms capable of handling the computational complexities of machine learning workflows on large data sets •           Write code that provides reliable automated data pipelines, included automated quality checks •           Take ownership of the data engineering code base, focusing on high-quality delivery of both code and data •           Be a technical leader by building pipelines to explore and leverage alternative non-traditional data sources   Qualifications: •           1-8 years’ relevant experience and a degree in Computer Science (or related field) with a strong GPA •           Strong foundational skills in SQL, query performance tuning, and bash/shell scripting •           Expert-level programming skills in Python and proficient with one of the following: Java/Scala, C/C++ •           Expertise in data modeling and choosing an appropriate model given the situation •           Capital Markets and Asset Management experience and related datasets/vendors is a plus •           Ability to provide thought leadership and contribute to the firm's overall technology strategy

Position Summary: The Data Engineer is a technical leader and hands-on developer responsible for designing, building, and optimizing data pipelines and infrastructure to support analytics and reporting. This role will serve as the lead developer on strategic data initiatives, ensuring scalable, high-performance solutions are delivered effectively and efficiently. The ideal candidate is self-directed, thrives in a fast-paced project environment, and is comfortable making technical decisions and architectural recommendations. The ideal candidate has prior experience in modern data platforms, most notable Databricks and the “lakehouse” architecture. They will work closely with cross-functional teams, including business stakeholders, data analysts, and engineering teams, to develop data solutions that align with enterprise strategies and business goals. Experience in the financial industry is a plus, particularly in designing secure and compliant data solutions.Responsibilities:·  Design, build, and maintain scalable ETL/ELT pipelines for structured and unstructured data.·  Optimize data storage, retrieval, and processing for performance, security, and cost-efficiency.·  Ensure data integrity and governance by implementing robust validation, monitoring, and compliance processes.·  Consume and analyze data from the data pipeline to infer, predict and recommend actionable insight, which will inform operational and strategic decision making to produce better results.·  Empower departments and internal consumers with metrics and business intelligence to operate and direct our business, better serving our end customers.·  Determine technical and behavioral requirements, identify strategies as solutions, and section solutions based on resource constraints.·  Work with the business, process owners, and IT team members to design solutions for data and advanced analytics solutions.·  Perform data modeling and prepare data in databases for analysis and reporting through various analytics tools.·  Play a technical specialist role in championing data as a corporate asset.·  Provide technical expertise in collaborating with project and other IT teams, internal and external to the company.·  Contribute to and maintain system data standards.·  Research and recommend innovative, and where possible automated approaches for system data administration tasks. Identify approaches that leverage our resources and provide economies of scale.·  Engineer system that balances and meets performance, scalability, recoverability (including backup design), maintainability, security, high availability requirements and objectives. Skills: ·  Databricks and related – SQL, Python, PySpark, Delta Live Tables, Data pipelines, AWS S3 object storage, Parquet/Columnar file formats, AWS Glue.·  Systems Analysis - The application of systems analysis techniques and procedures, including consulting with users, to determine hardware, software, platform, or system functional specifications.·  Time Management - Managing one's own time and the time of others. ·  Active Listening - Giving full attention to what other people are saying, taking time to understand the points being made, asking questions as appropriate, and not interrupting at inappropriate times. ·  Critical Thinking - Using logic and reasoning to identify the strengths and weaknesses of alternative solutions, conclusions or approaches to problems. ·  Active Learning - Understanding the implications of new information for both current and future problem-solving and decision-making. ·  Writing - Communicating effectively in writing as appropriate for the needs of the audience. ·  Speaking - Talking to others to convey information effectively. ·  Instructing - Teaching others how to do something. ·  Service Orientation - Actively looking for ways to help people. ·  Complex Problem Solving - Identifying complex problems and reviewing related information to develop and evaluate options and implement solutions. ·  Troubleshooting - Determining causes of operating errors and deciding what to do about it. ·  Judgment and Decision Making - Considering the relative costs and benefits of potential actions to choose the most appropriate one. Experience and Education: ·  High School Diploma (or GED or High School Equivalence Certificate).·  Associate degree or equivalent training and certification.·  5+ years of experience in data engineering including SQL, data warehousing, cloud-based data platforms. ·  2+ years Project Lead or Supervisory experience preferred.  IQ Ventures is an Equal Opportunity Employer. It’s our policy is not to discriminate against any applicant or employee based on actual or perceived race, age, sex or gender (including pregnancy), marital status, national origin, ancestry, citizenship status, mental or physical disability, religion, creed, color, sexual orientation, gender identity or expression (including transgender status), veteran status, genetic information, or any other characteristic protected by applicable federal, state or local law. We will provide accommodations to applicants needing accommodations to complete the application process. At this time, IQ Ventures cannot transfer nor sponsor a work visa for this position. Applicants must be authorized to work directly for any employer in the United States without visa sponsorship. IQ Ventures is an established, profitable technology and financial services company serving clients nationwide. Join the IQ Ventures team in our bright, modern Dublin, OH offices. IQ Ventures is well-regarded for its high integrity and collaborative leadership culture that rewards both individual thinking and team decision-making. Our leadership team is comprised of seasoned professionals who bring their vast experience and high standards of excellence to their work. We are not accepting candidates from third-party recruiters at this time.

Data Engineer III Job ID # 81627Rate type: W2 only Please note: The selected candidate will be required to report to the office two times per week at one of the following locations: Montvale, NJ; Denver, CO; Atlanta, GA; or Orlando, FL. Additionally, at least one interview will be conducted in person. Proven experience as a Data Engineer, with a strong focus on Azure Synapse Analytics. Proficiency in Python for data manipulation, scripting, and automation. Expertise in Azure services, including Azure Data Factory, Azure SQL Database, and Azure Storage. Hands-on experience with Azure AI Document Intelligence for document processing and data extraction. Create, manage, and optimize ETL processes to ensure efficient data flow and high data quality. Strong understanding of data warehousing concepts, ETL processes, and data modeling techniques. Monitor and troubleshoot data systems, ensuring optimal performance and reliability. Implement best practices for data security, metadata management, and compliance with organizational policies. Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment. Effective communication and interpersonal skills. Python (more than 3 years' experience with libraries to extract data from PDF, MS office files). Azure Synapse to create data Python pipeline jobs and clusters. Good knowledge of API integration

POSITION: Sr. Data EngineerCLIENT: Fortune 150 Company; Financial ServicesSUMMARY DESCRIPTION:The Data Engineer will serve in a strategic role designing and managing the infrastructure that supports data storage, transforming, processing, and retrieval enabling efficient data analysis and decision-making within the organization. This position is critical as part of the Database and Analytics team responsible for design, development, and implementation of complex enterprise-level data integration and consumption solutions. It requires a highly technical, self-motivated senior engineer who will work with analysts, architects, and systems engineers to develop solutions based on functional and technical specifications that meet quality and performance requirements.PRIMARY DUTIES AND RESPONSIBILITIES:<ul><li>Utilize experience in ETL tools, with at least 5 years dedicated to Azure Data Factory (ADF), to design, code, implement, and manage multiple parallel data pipelines. Experience with Microsoft Fabric, Pipelines, Mirroring, and Data Flows Gen 2 usage is required.</li><li>Apply a deep understanding of data warehousing concepts, including data modeling techniques like star and snowflake schemas, SCD Type 2, Change Data Feeds, Change Data Capture. Also demonstrates hands-on experience with Data Lake Gen 2, Delta Lake, Delta Parquet files, JSON files, big data storage layers, optimize and maintain big data storage using Partitioning, V-Order, Optimize, Vacuum and other techniques.</li><li>Design and optimize medallion data models, warehouses, architectures, schemas, indexing, and partitioning strategies.</li><li>Collaborate with Business Insights and Analytics teams to understand data requirements and optimize storage for analytical queries.</li><li>Modernize databases and data warehouses and prepare them for analysis, managing for optimal performance.</li><li>Design, build, manage, and optimize enterprise data pipelines ensuring efficient data flow, data integrity, and data quality throughout the process.</li><li>Automate efficient data acquisition, transformation, and integration from a variety of data sources including databases, APIs, message queues, data streams, etc.</li><li>Competently performs advanced data tasks with minimal supervision, including architecting advanced data solutions, leading and coaching others, and effectively partnering with stakeholders.</li><li>Interface with other technical and non-technical departments and outside vendors on assigned projects.</li><li>Under the direction of the IT Management, will establish standards, policies and procedures pertaining to data governance, database/data warehouse management, metadata management, security, optimization, and utilization.</li><li>Ensure data security and privacy by implementing access controls, encryption, and anonymization techniques as per data governance and compliance policies.</li><li>Expertise in managing schema drift within ETL processes, ensuring robust and adaptable data integration solutions.</li><li>Document data pipelines, processes, and architectural designs for future reference and knowledge sharing.</li><li>Stay informed of latest trends and technologies in the data engineering field, and evaluate and adopt new tools, frameworks, and platforms (like Microsoft Fabric) to enhance data processing and storage capabilities.</li><li>When necessary, implement and document schema modifications made to legacy production environment.</li><li>Perform any other function required by IT Management for the successful operation of all IT and data services provided to our clients.</li><li>Available nights and weekends as needed for system changes and rollouts.</li></ul>EDUCATION AND EXPERIENCE REQUIREMENTS:<ul><li>Bachelor’s or Master’s degree in computer science, information systems, applied mathematics, or closely related field.</li><li>Minimum of ten (10) years full time employment experience as a data engineer, data architect, or equivalent required.</li></ul>SKILLS:<ul><li>Experience in working with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures, and integrated datasets using traditional and modern data integration technologies (such as ETL, ELT, MPP, data replication, change data captures, message-oriented data movement, API design, stream data integration and data virtualization)</li><li>Experience working with cloud data engineering stacks (specifically Azure and Microsoft Fabric), Data Lake, Synapse, Azure Data Factory, Databricks, Informatica, Data Explorer, etc.</li><li>Strong, in-depth understanding of database architecture, storage, and administration utilizing Azure stack.</li><li>Deep understanding of Data architectural approaches, Data Engineering Solutions, Software Engineering principles and best practices.</li><li>Working knowledge and experience with modern BI and ETL tools (Power BI, Power Automate, ADF, SSIS, etc.)</li><li>Experience utilizing data storage solutions including Azure Blob storage, ADLS Gen 2.</li><li>Solid understanding of relational and dimensional database principles and best practices in a client/server, thin-client, and cloud computing environment.</li><li>Advanced working knowledge of TSQL and SQL Server, transactions, error handling, security and maintenance with experience writing complex stored procedures, views, and user-defined functions as well as complex functions, dynamic SQL, partitions, CDC, CDF, etc.</li><li>Experience with .net scripting and understanding of API integration in a service-oriented architecture.</li><li>Knowledge of reporting tools, query language, semantic models with specific experience with Power BI.</li><li>Understanding of and experience with agile methodology.</li><li>PowerShell scripting experience desired.</li><li>Experience with Service Bus, Azure Functions, Event Grids, Event Hubs, Kafka would be beneficial.</li><li>Experience working in Agile methodology.</li></ul>Working Conditions:<ul><li>Available to work evenings and/or weekends (as required).</li><li>Workdays and hours are Monday through Friday 8:30 am to 5:30 pm ET.</li></ul>

Position OverviewLocal company has an opening for a Microsoft SQL Developer focused on Business Intelligence (BI) and Data Transformation. This role is ideal for someone with a strong SQL development background who also has experience with BI tools and data transformation to support data driven decision making. As an MS SQL Developer, you will design, develop, and maintain databases, write SQL queries and stored procedures, optimize database performance, and create custom reports while ensuring data integrity and security. You will also work with Business Intelligence (BI) tools such as Tableau and play a key role in data transformation to enhance reporting, analytics, and business decision-making.<ul><li>ResponsibilitiesDatabase Design and Development:</li><li>Designing and implementing efficient database structures.</li><li>Creating and modifying database tables, views, stored procedures, and functions.</li><li>Developing and maintaining database schemas to support business requirements.</li><li>SQL Development:</li><li>Writing and optimizing SQL queries for data retrieval, transformation, and manipulation.</li><li>Developing and maintaining stored procedures, triggers, and functions to streamline processes.</li><li>Ensuring SQL code is efficient, readable, and well-documented.</li><li>Database Performance and Optimization:</li><li>Identifying and resolving performance bottlenecks in database queries and applications.</li><li>Implementing indexing, partitioning, and other optimization techniques.</li><li>Monitoring and fine-tuning database performance to enhance efficiency.</li><li>Data Integrity, Security, and Transformation:</li><li>Implementing and maintaining database security measures.</li><li>Ensuring data integrity, consistency, and quality throughout data pipelines.</li><li>Developing and managing ETL (Extract, Transform, Load) processes for data integration and transformation.</li><li>Performing data cleansing, validation, and standardization to support analytics and reporting.</li><li>Business Intelligence & Reporting:</li><li>Developing and maintaining custom reports using</li><li>SQL Server Reporting Services (SSRS)</li><li>or other BI tools.</li><li>Creating interactive dashboards and visualizations using</li><li>Tableau:</li><li>or similar BI platforms.</li><li>Analyzing data trends and providing actionable insights to support business decisions.</li><li>Collaboration & Communication:</li><li>Working closely with business analysts, data engineers, and other developers to understand data needs.</li><li>Translating business requirements into efficient database solutions.</li><li>Communicating technical concepts clearly to both technical and non-technical stakeholders.</li><li>Troubleshooting & Problem-Solving:</li><li>Identifying and resolving database errors and performance issues.</li><li>Troubleshooting SQL queries, stored procedures, and BI reports.</li><li>Providing technical support to users and assisting with data-related inquiries.</li><li>RequirementsA Bachelor’s Degree in a closely related field is desired</li><li>A minimum of 3-5 years of related experience is required</li><li>The following certifications are strongly preferred: Azure Data Fundamental</li><li>Azure Database Administrator</li><li>Tableau Desktop Associate</li><li>Demonstrated proficiency in the following competencies is required:</li><li>Proficiency in SQL: Strong understanding of SQL syntax, data manipulation, and database concepts.</li><li>Experience with Microsoft SQL Server: Familiarity with SQL Server database management system, including SQL Server Management Studio (SSMS).</li><li>Database Design and Modeling: Experience in designing and implementing relational databases.</li><li>Query Optimization: Knowledge of query optimization techniques and performance tuning.</li><li>Stored Procedures and Functions: Expertise in developing and maintaining stored procedures, triggers, and functions.</li><li>ETL & Data Transformation: Experience with ETL tools and techniques for data extraction, transformation, and loading.</li><li>Business Intelligence (BI) Tools: Hands-on experience with</li><li>Tableau, Power BI, or other BI tools</li><li>for data visualization and reporting.</li><li>Troubleshooting and Problem-Solving: Ability to diagnose and resolve database and performance-related issues.</li><li>Previous experience working with ERP systems is required - Experience with Acumatica is a plus</li><li>Must have excellent communication skills and the ability to convey technical information effectively to diverse audiences</li><li>Must thrive in a team environment and be able to work collaboratively with cross-functional teams</li><li>Proven ability to be flexible with changing priorities is a must</li><li>Must have the ability to work a minimum 40 hours as scheduled and maintain a punctual, reliable attendance record</li></ul>

Company Overview:ALT Sports Data is at the forefront of transforming fan engagement through innovative, data-driven solutions. Our mission is to deliver advanced sports betting odds and markets that connect fans with sports in exciting new ways. We are seeking a skilled Data Engineer to design and maintain robust data pipelines that power our analytics and betting platforms. If you’re passionate about building scalable data systems and have an interest in sports betting, this role is your chance to make an impact! Key Responsibilities:<ul><li>Design, develop, and maintain scalable data pipelines and architectures to support complex data processing and transformation needs.</li><li>Collaborate with cross-functional teams, including data scientists, analysts, and product managers, to integrate data solutions seamlessly with existing systems.</li><li>Optimize data processing workflows for performance, reliability, and scalability, adhering to data engineering best practices.</li><li>Implement data quality checks, validation procedures, and monitoring to ensure data integrity and accuracy across the pipeline.</li><li>Develop and manage ETL processes to aggregate data from diverse sources, ensuring timely and accurate data delivery for analytics and betting applications.</li><li>Monitor and troubleshoot data systems, proactively addressing issues to maintain consistent data flow and availability.</li><li>Stay current with industry trends and emerging technologies, continuously improving data engineering practices and infrastructure.</li></ul>Education & Experience & Skills:<ul><li>Bachelor’s or Master’s degree in Mathematics, Statistics, Computer Science, or a related quantitative/technical field.</li><li>3+ years of experience in data engineering or DevOps roles.</li><li>Advanced proficiency in Python.</li><li>Expertise in SQL and experience with relational databases like PostgreSQL, MySQL, or similar.</li><li>Familiarity with big data technologies such as Apache Spark, Hadoop, or Kafka (preferred).</li><li>Experience building data pipelines using AWS or Django frameworks.</li><li>A passion for sports and a willingness to learn about new sports or data technologies.</li></ul>Personal Attributes:<ul><li>Strong problem-solving skills with the ability to tackle complex data engineering challenges.</li><li>Meticulous attention to detail with a focus on data accuracy and system reliability.</li><li>Exceptional communication skills to collaborate with analysts, developers, and betting specialists.</li><li>High integrity in handling sensitive and proprietary betting data.</li><li>A proactive learner with enthusiasm for adopting new tools and methodologies.</li></ul>Compensation:<ul><li>Competitive market-based salary.</li><li>Comprehensive benefits, including medical, dental, and a 401(k) match.</li><li>Opportunity to share in company equity.</li><li>Performance-based bonuses tied to individual and company success.</li></ul>Why Join ALT Sports Data?<ul><li>Join a dynamic team driving innovation in sports analytics and betting infrastructure.</li><li>Work on high-impact projects that power fan engagement through robust data systems.</li><li>Enjoy a flexible, hybrid work environment that supports work-life balance and encourages innovation.</li><li>ALT Sports Data is an equal opportunity employer dedicated to fostering a diverse and inclusive workplace. We welcome applications from all qualified candidates, valuing the unique perspectives they bring.</li></ul>

Zaddy Solutions is performing this search on behalf of a client. We're partnering with a rapidly growing organization in the healthcare innovation space to find a Senior Data Engineer to join their dynamic team. This is an exciting opportunity to lead the design, planning, and development of cutting-edge data solutions, playing a pivotal role in transforming raw data into actionable insights. If you're a hands-on leader with a passion for building robust data pipelines and driving technological advancements, we encourage you to apply! What You'll Do:<ul><li>Design, develop, and implement reliable and effective data solutions based on evolving business requirements.</li><li>Lead the design and development of complex data transformations and contribute to data architecture discussions.</li><li>Identify trends in data sets and develop algorithms to enhance data utility.</li><li>Build and maintain optimal data pipeline architectures, ensuring adherence to security standards.</li><li>Drive internal process improvements through automation, optimization, and infrastructure redesign for scalability.</li><li>Establish best practices for database design and development activities.</li><li>Create intricate functions, scripts, and services to support the Data Services team.</li><li>Conduct code reviews and mentor junior engineers, fostering a collaborative and high-performing environment.</li><li>Serve as a subject matter expert on key business projects, providing detailed analysis and recommendations.</li><li>Analyze and integrate new technologies to improve application design and functionality.</li><li>Develop and automate solutions to consume data from diverse sources, including external APIs.</li><li>Program and modify code in languages like Java, JSON, Python, and Spark to support Data Warehouse solutions.</li><li>Design and deploy enterprise-scale cloud infrastructure solutions, with a focus on Azure implementations.</li></ul>What You'll Bring:<ul><li>Minimum of eight (8) years of experience in provisioning, configuring, and developing solutions within Azure Data Lake, Azure Data Factory, Azure SQL Data Warehouse, Azure Synapse, and Cosmos DB.</li><li>Eight (8) years of experience implementing software development methodologies.</li><li>Eight (8) years of experience working with relational databases, including building and optimizing big data pipelines and architectures.</li><li>Proven experience performing root cause analysis on internal and external data and processes.</li><li>Proficiency with Source Control and management tools such as Azure DevOps and Gitlab/GitHub.</li><li>Experience transforming requirements into Design Concepts and ERDs using tools like Visio.</li><li>Minimum of five (5) years of hands-on experience with cloud orchestration and automation tools, and CI/CD pipeline creation.</li><li>Bachelor’s degree in a quantitative discipline such as Computer Science, Statistics, Mathematics, or Engineering from an accredited institution.</li><li>Strong knowledge of DevOps principles and practices.</li><li>Fluency in at least one scripting or programming language such as Python, Java, or JSON.</li><li>Understanding of applicable data privacy practices and laws.</li><li>Familiarity with common SDLC models (Waterfall, Agile - Scrum and Kanban).</li><li>Experience with non-relational database (NoSQL) designs (e.g., MongoDB) and relational databases (e.g., MS SQL Server).</li><li>Knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.</li></ul>

Key Responsibilities: <ul><li>Design, develop, and maintain scalable and robust data pipelines.</li><li></li><li>Integrate data from multiple sources, including APIs, cloud platforms, databases, and third-party systems.</li><li></li><li>Build and manage data infrastructure and architecture that supports analytics and reporting needs.</li><li></li><li>Ensure data quality, consistency, reliability, and governance across systems.</li><li></li><li>Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.</li><li></li><li>Implement data security best practices to ensure compliance with privacy and regulatory standards (GDPR, HIPAA, etc.).</li><li></li><li>Monitor pipeline performance and troubleshoot issues.</li><li></li><li>Automate repetitive tasks to improve efficiency and reduce manual workloads.</li></ul><li></li> Required Skills and Qualifications: <ul><li>Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or related field.</li><li></li><li>Experience in data engineering or a related role.</li><li></li><li>Proficiency in SQL and at least one programming language (e.g., Python, Scala, Java).</li><li></li><li>Experience with data pipeline tools such as Apache Airflow, Luigi, or similar.</li><li></li><li>Strong understanding of ETL/ELT processes and tools (e.g., Talend, Informatica, Fivetran, dbt).</li><li></li><li>Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.</li><li></li><li>Experience working with big data technologies like Spark, Hadoop, or Kafka.</li><li></li><li>Familiarity with data warehousing solutions (e.g., Snowflake, Redshift, BigQuery).</li><li></li><li>Strong problem-solving and communication skills.</li></ul><li></li>

Primary Responsibilities:· Works with business stakeholders to build interactive and creative dashboards· Seeks out creative solutions to problems and opportunities for continuous improvement across all BI reporting platforms· Good understanding of custom visualizations available for different reporting tools· Work with data teams to understand the different data marts and build models for reporting· Write complex SQL queries to do analysis on the data· Build pipelines using ADF/Talend for data ingestion· Build small scale Power App applications to meet requirements from business teams Minimum Experience:· 10 years of professional experience in a Business Analytics or Business reporting type capacity Knowledge, Skills & Abilities:· Strong communication skills: ability to interact with technical teams, communicate technical information to non-technical audiences and gain buy-in from management· Proven technical skills and understanding of IT concepts. Requires proficiency in Microsoft Power BI & Tableau· Proficiency with data querying or modeling technique with SQL· Experience with Business Intelligence technologies like Power BI, ADF etc.· Strong understanding of different visualizations available in reporting tools· Strong understanding of database design concepts· Coding proficiency Python and/or R· Ability to operate successfully and independently in a fast-paced environment· Be comfortable with ambiguity and curious to learn new skills

About the Role We are seeking a Data Engineer to design, implement, and optimize a modern cloud-based data platform using Google BigQuery and GCP-native tools. This role will be responsible for transforming data into high-quality, structured datasets to enable self-service analytics in Tableau and other BI tools. You will ensure that our BigQuery data warehouse is scalable, cost-efficient, and aligned with business intelligence needs.  Key Responsibilities BigQuery Data Warehouse Management and Operations <ul><li>Design and implement scalable data pipelines using GCP-native tools </li><li>Develop real-time and batch data pipelines using Dataflow, Apache Beam, and BigQuery Pub/Sub for streaming and structured data ingestion. </li><li>Optimize performance with BigQuery partitioning, clustering, materialized views, and optimized SQL transformations. </li><li>Automate and schedule workflows with tools like dbt/Dataform, Airflow/Composer, and/or Cloud Workflows. </li><li>Define and manage fact tables (transactions, events, KPIs) and dimension tables (customers, providers, hospitals, products, locations). </li></ul>Streaming & Real-Time Analytics <ul><li>Develop streaming ingestion pipelines using Dataflow (Apache Beam), Pub/Sub. </li><li>Enable event-driven transformations for real-time data processing. </li><li>Performance optimizations for real-time dashboards in Tableau, Looker, or Data Studio, for both our compute [ financial ] costs and dashboard-user experience. </li></ul>Data Governance, Quality & Security <ul><li>Implement schema validation, deduplication, anomaly detection, and reconciliation across multiple sources. </li><li>Define access controls, row-level security (RLS), and column-level encryption to ensure data protection, compliance. </li><li>Maintain data lineage and metadata tracking using tools like OpenLineage, Dataplex Catalog. </li></ul>Optimize & Automate Data Pipelines <ul><li>Develop incremental data refresh strategies to optimize cost and performance. </li><li>Automate data transformation workflows with dbt, Dataform, Cloud Composer (Apache Airflow), and Python. </li><li>Monitor pipeline performance and cloud cost efficiency with Cloud Logging, Monitoring, and BigQuery BI Engine. </li></ul>Enable Self-Service BI & Analytics <ul><li>Ensure that tables and views are structured for fast and efficient queries in Tableau, Looker, and self-service BI tools. </li><li>Work with data analysts to optimize SQL queries, views, and datasets for reporting. </li><li>Provide data documentation and best practices to business teams for efficient self-service analytics. </li><li>Collaborate with data producers to ensure data is well understood at product time, and ahead of ingest.   </li><li>Curate and maintain data dictionaries, data catalog, so users can understand what they are accessing. </li></ul> Required Qualifications Experience in Data Architecture & Engineering <ul><li>2+ years of experience in analytics/data engineering, cloud data architecture, or ELT development. </li><li>Strong hands-on experience with SQL, and cloud-based data processing. </li><li>Hands-on Development experience with Python [ or other programming language(s) ]. </li></ul>Expertise in GCP & BigQuery Data Processing <ul><li>Deep understanding of ELT/ETL principles </li><li>Proficiency in dbt, Dataform, or SQL-based transformation tools for data modeling. </li><li>Experience with GCP services: BigQuery, Dataflow (Apache Beam), Pub/Sub, Cloud Storage, and Cloud Functions. </li></ul>BigQuery Optimization & Performance Tuning <ul><li>Experience optimizing BigQuery partitioning, clustering, materialized views, and query performance. </li><li>Expertise in cost-efficient query design and workload optimization strategies. </li></ul>Experience in Streaming & Real-Time Processing <ul><li>Hands-on experience with streaming data pipelines using Dataflow (Apache Beam), Apache Flink, Pub/Sub, or Kafka. </li><li>Familiarity with real-time data transformations and event-driven architectures. </li></ul>Experience Supporting BI & Analytics <ul><li>Strong knowledge of Tableau, Looker, and BI tools, ensuring reporting is optimized </li><li>Ability to collaborate with data analysts and business teams to define data models and metrics. </li></ul> Bonus Skills (Preferred but Not Required) <ul><li>Knowledge of Cloud Composer (Apache Airflow) for data orchestration. </li><li>Familiarity with AI/ML model deployment and machine learning pipelines in GCP Vertex AI, Jupyter Notebooks, Pandas, etc. </li><li>Understanding of and experience with development/deployment patterns and dependency managementt, CI/CD, Testing, CodeQuality, Devcontainers or nixpkgs, poetry/uv </li><li>Programming abilities beyond python: Golang and/or Java/Kotlin/JVM  </li><li>Database Administration, experience with varied database systems [ NoSQL, graph, etc ]. </li><li> </li></ul>Why Join Us? <ul><li>Work on a next-generation data platform built on Google BigQuery and GCP-native tools. </li><li>Drive real-time data processing and self-service BI enablement in Tableau, Looker, and advanced analytics. </li><li>Work with modern cloud-based technologies such as BigQuery, dbt, Dataflow, and Cloud Functions. </li><li>Fully remote opportunity with a high-impact data engineering role. </li></ul>

Epic! Data Engineer Interview Guide

Overview

Epic! Data Engineer Interview Process

Epic! Data Engineer Interview Questions

Epic! Data Engineer Jobs