LEAD DATA ENGINEERSAN FRANCISCO BAY AREA (1-2 days Onsite)$200,000 - $220,000 BASE + BONUS & EQUITY THE COMPANY:This wellness startup company focuses on developing supplements and probiotics aimed at enhancing gut health, improving digestion, and supporting metabolic balance. Leveraging scientific research and personalized data insights, the company tailors its formulations to meet individual health needs. THE ROLE:This role will design, build, and optimize the infrastructure required for seamless data flow and analytics. The role involves managing scalable data pipelines, supporting ML initiatives, and ensuring data integrity, making it ideal for a hands-on engineer who thrives in a fast-paced, high-impact environment. <ul><li>Design, build, and maintain scalable ETL pipelines using tools like DBT, Fivetran, and Airflow</li><li>Optimize and manage data warehouse and lakehouse environments on platforms like Snowflake, ensuring data availability, reliability and performance</li><li>Implement and manage scalable workflows using Docker, Kubernetes, and other systems to automate data processing </li><li>Collaborate with other teams to support ML model deployment in production</li><li>Ensure data compliance with standards and regulations (GDPR, CCPA), manage data lineage, and maintain metadata for secure data handling practices </li><li>Develop and execute processes for data validation, cleansing, and anomaly detection for data quality assurance </li></ul> YOUR SKILLS AND EXPERIENCE:<ul><li>5+ YOE in data engineering</li><li>Start-Up or Retail industry experience</li><li>Advanced SQL and Python skills</li><li>Experience working with a modern data stack (ex: Snowflake, Fivetran, dbt, airflow, etc)</li><li>Strong experience with cloud platforms (GCP, AWS, Azure) and tech like Hive, Postgres, Redis</li><li>Proven experience working autonomously and spearheading projects with minimal oversight</li></ul>(Experience with Kafka, Spark, Kubernetes a plus) MUST BE AUTHORIZED TO WORK IN THE US

Senior Data Engineer Amicus is currently working with an exciting start-up in the business intelligence space, who are looking to bolster their growing engineering team with the addition of an experienced Senior Data Engineer. Requirements: <ul><li>3-5 years’ experience as a data engineer.</li><li>Experience with Python, Apache Airflow and GCP.</li><li>Experience building ETL pipelines.</li><li>Experience in an early-stage startup environment.</li><li>Must be located in the San Francisco Bay Area and be willing to come on-site 2 days per week.</li><li>Must be a US Citizen or Greencard Holder.</li></ul> Benefits: <ul><li>$180K-$250K base salary (DOE).</li><li>Generous stock options.</li><li>Health insurance.</li></ul> To be considered for this position, please send your resume to harry.tyler@amicusjobs.com.

Title: Data Engineer(Databricks Azure)Location: San Francisco, CA(Remote)Contract ResponsibilitiesLead, design and implement innovative analytical solution using Tableau, SQL, and other Big Data related technologiesWork with product and engineering team to understand requirements, evaluate new features and architecture to help drive decisionsBuild collaborative partnerships with architects, technical leads and key individuals within other functional groupsPerform detailed analysis of business problems and technical environments and use this in designing quality technical solutionActively participate in code review and test solutions to ensure it meets best practice specificationsBuild and foster a high-performance engineering culture, mentor team members and provide team with the tools and motivation to make things happenWork with stakeholders and cross-functional teams to develop new solutions or enhance existing solutionDemonstrate our xxx values of Passion for Client Service, Innovation, Expertise, Balance, Respect for All, Teamwork, and InitiativeWe’re excited about you if you have: 8-10 years of software development and deployment experience with at least 5 years of hands-on experience with SQL, Databricks, ADF, Datastage (or other ETL tool), SSAS cubes, Cognos, Tableau, Thoughtspot and other BI tools Write SQL for processing raw data, kafka ingestions, adf pipelines, data validation and QA Knowledge working with APIs to collect or ingest data Strong Database knowledge, SQL & No-SQL preferred Experience building data ingestion pipelines (simulating Extract, Transform, Load workload), data warehouse or database architectureExperience writing design documentation, Source to Target mapping documentation, manage confluence pages.Experience in converting business functionalities into technical jira storiesExperience in analyzing huge datasets, identify trends, patterns, and outliers to extract meaningful insightsStrong experience with data modeling, design patterns, building highly scalable Business Intelligence Solutions and distributed applicationsKnowledge of cloud platforms, for example: Experience with Azure, AWS or equivalent cloud platformsExperience with storing, joining, filtering, and analyzing data using SQL, Spark, Hive etc.Experience working with continuous integration framework, building regression-able code within data world using GitHub, Jenkins and related applicationsExperience with programming/scripting languages such as Scala/Java/Python/R etc. (any combination)Analytical approach to problem-solving with an ability to work at an abstract level and gain consensus; excellent interpersonal, leadership and communicationData-oriented personality. Motivated, independent, efficient and able to handle several projects; work under pressure with a solid sense for setting prioritiesAbility to work in a fast-paced (startup like) agile development environment Friendly, articulate, and interested in working in a fun, small team environment

The Senior Data Engineer will design, build, and implement data integration solutions, including data pipelines, data APl's, and ETL jobs, to meet the data needs of applications, services, data assets, and business intelligence and analytical tools. Working with business users, data architects, application development teams, data analytics teams, business analytics, product managers, and the data governance COE, the Senior Data Engineer will design and develop ELT between databases, data assets, external partners, and third-party systems in a combination of cloud and on-premises platforms. How will you make an impact:<ul><li>Develops and maintains scalable data pipelines and builds out new integrations to support continuing increases in data volume and complexity</li><li>Designs and develops scalable ELT packages/pipelines for data extract load and transform data between source systems, extract on, and integration of data into various data assets, including data lake and fit for purpose data repositories, both on prem and cloud</li><li>Collaborate with data architects to understand data structures and concepts including conceptual, logical, and dimensional models and provide early feedback </li><li>Designs and develops data migrations in support of enterprise application and system implementations from legacy systems</li><li>Writes functional specifications for data pipelines and APls and writes and performs unit/integration tests Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues</li><li>Assists in planning, coordinating, and executing engineer ng projects</li><li>Supports and collaborates with other Engineers through evaluation, design analysis, and development phases</li><li>Maintains knowledge, ensures competency and compliance with policies and procedures, to be the technical expert while collaborating with cross-functional teams </li><li>This list is not all inclusive and you are expected to perform other duties as requested or assigned</li></ul> Skill/Experience<ul><li>Bachelor’s degree in computer science, Mathematics, Statistics, or other related technical field</li><li>5+ years of experience in ETL tool, cloud technologies like Azure Data Factory, Informatica</li><li>3+ years of experience in Matillion and Snowflake</li><li>Experience in data profiling, data analysis and building Data Quality metrics</li><li>Experience in reference data management tools like EBX, Informatica MOM or related tools</li><li>Experience building ETL for moving the data from desperate source system into data lake and dimensional models</li><li>Expertise in SQL(No-SOL experience is a plus) against relational and cloud data structures</li><li>Experience in data virtualization, Tibco Data Virtualization preferred</li><li>Experience in documenting and implementing the best practices for building data transformation and load processes.</li><li>Knowledge of best practices and IT operations in an a ways-up, always-available service</li><li>Experience with or knowledge of Agile Software Development methodologies</li><li>Excellent problem solving and troubleshooting skills</li><li>Process oriented with great documentation skills</li><li>Excellent oral and written communication skills with a keen sense of customer service</li></ul>

Title: Data EngineerLocation: San Francisco, CA (1-2 Days onsite/Week)Duration: 12+ Month Contract Key Responsibilities:• Design and implement scalable data pipelines using Databricks and Spark.• Develop and maintain data processing applications in Scala and Python.• Integrate and manage data streams using Kafka.• Build and maintain microservices using Spring Boot.• Collaborate with data scientists and analysts to understand data requirements and deliver solutions.• Optimize and troubleshoot data workflows to ensure high performance and reliability.• Ensure data quality and integrity across various data sources. Required Skills and Qualifications:• Bachelor’s degree in Computer Science, Engineering, or a related field.• Proven experience as a Data Engineer or in a similar role.• Proficiency in Databricks, Spark, Scala, Kafka, Python, and Spring Boot.• Strong understanding of microservices architecture.• Experience with cloud platforms (e.g., AWS, Azure, GCP) is a plus.• Excellent problem-solving skills and attention to detail.• Strong communication and teamwork abilities.

Company Description Promoted (YC W21) is a company that unifies organic search, promotions, and ads in one optimized system for scaled marketplaces. They specialize in sorting marketplace search results to promote the best listings and increase revenue. The company is based in the San Francisco Bay Area and offers flexibility for remote work. Role Description This is a full-time hybrid role for a Senior Data Engineer at Promoted.ai. The Senior Data Engineer will be responsible for tasks such as data engineering, data modeling, ETL processes, data warehousing, and data analytics. The role is primarily located in the San Francisco Bay Area with the option for some remote work. Qualifications <ul><li>Data Engineering, ETL, Data Warehousing skills</li><li>Data Modeling and Data Analytics skills</li><li>Experience in working with large datasets</li><li>Proficiency in SQL and Python</li><li>Strong problem-solving and analytical skills</li><li>Ability to work independently and collaboratively in a team</li><li>Experience with cloud platforms such as AWS or GCP</li><li>Master's degree in Computer Science, Engineering, or related field</li></ul>

Data Engineer - SF - onsite $150k - $275k AI start up in SF - responsible for building in the AI agents space - 20mil series A. We’re looking for a Data Engineer to be our first data hire. You’ll play a key role in shaping our product and driving growth with data. What You’ll Do<ol><li>Build From Scratch: Design and own our ETL pipeline and data warehouse.</li><li>Analyze Interactions: Use data to improve how users interact with our AI agents.</li><li>Refine Our Product: Help us make data-driven decisions to improve user experiences.</li><li>Support Growth: Identify ways to boost adoption and satisfaction.</li></ol> About You<ul><li>3+ years of experience in data science / data engineering or a related role.</li><li>Skilled in Python, data pipelines, and tools like Snowflake or similar.</li><li>Able to turn data into actionable insights and clear recommendations.</li><li>Comfortable taking ownership and building systems from scratch.</li></ul> Why It Might Not Be for You<ul><li>We expect high performance and innovation.</li><li>This role is fully in-person.</li></ul>

Pay Rate: $80/hr to $90/hr on W2 Type: Contract Must Haves:<ul><li>Java</li><li>Python</li><li>Shell</li></ul> Responsibilities: -Design, develop, and manage data pipelines and workflows to enable efficient and accurate data processing using Trino SQL/Spark SQL warehoused in HDFS datasets. -Effectively performs code designs and reviews/approves test cases. -Implement data quality checks and audits to maintain high data accuracy and integrity. -Produces elegant and efficient designs, high performance, and scalable code that allows for easy extension to future needs. -Collaborate with cross-functional teams, especially data engineering, to understand data requirements and implement robust data solutions. -Work closely with data domain experts to gather data requirements, translate business needs into technical specifications, and communicate data insights effectively for sales representative workflow efficiency. -Optimize data storage for performance and scalability, ensuring efficient data Extraction, Transformation and Load (ETL). -Develop and maintain documentation related to data pipelines, QA, metrics, and data policy as it relates to best practice, compliance and GDPR. -Stay up to date with industry best practices and emerging trends in data engineering and analytics, including Generative AI as it impacts our data operations. Qualifications: -2+ years in using SQL and experience optimizing SQL databases for performance (Trino SQL, or Spark). -Demonstrated experience in managing data pipelines (like HDFS), data repository (like GitHub), workflows (like Apache Airflow), and ETL (best practice coding). -Ability to communicate complex technical concepts to both technical and non-technical individuals. -Experience working with multiple stakeholders, setting project priorities and delivering on Objectives and Key Results (OKRs). -Experience automating script changes in Python Preferred Qualifications: -BA/BS in engineering, computer science, or related technical field (such as statistics, or data science). -Excellent analytical skills, designing data workflows and analyzing data for anomalies, or setting data quality thresholds via automated solutions. -Familiarity with data governance principles -Program Manager experience -Demonstrated experience in managing data pipelines in HDFS -Experience running a scrum team and using Jira. -Spark SQL

We are looking for an experienced Cloud Data Engineer to join our direct client. Must be a US Citizen or a Green Card holder. The ideal candidate will have expertise in writing Python scripts, and crafting complex SQL queries along with AWS(EC2, S3 Redshift), Data Migration, Application Migration and ETL. Proficient in Cloud Data Migration, including transitioning on-premises applications to cloud environments. Responsibilities: <ul><li>Design, develop, modify, test, and automate data warehouse and business intelligence solutions, including architecture recommendations, quality management, metadata creation, troubleshooting, and performance tuning.</li><li>Develop transition and implementation plans, recommending enhancements to development, maintenance, and standards.</li><li>Contribute to the design, development, testing, implementation, and review of complex data warehouse and BI solutions.</li><li>Build and refine components of complex data warehouse applications, including software development from established requirements, report/dashboard creation, and prototype development.</li><li>Assist in designing technology infrastructure and configurations while recommending process improvements.</li><li>Review and implement patches and new versions of data warehouse applications, deploying code and software packages.</li><li>Play a key role in cross-functional initiatives and process improvement projects.</li><li>Exercise independent judgment in complex business decisions, ensuring high attention to detail, data integrity, and accuracy.</li><li>Communicate technical concepts clearly to non-technical audiences, demonstrating strong verbal and written communication skills.</li><li>Demonstrate broad knowledge of programming tools, design, implementation, and testing practices.</li><li>Rare domestic travel with potential overnight stays.</li></ul> Required Skills: <ul><li>Bachelor’s degree or equivalent experience.</li><li>Minimum of 8 years of relevant experience in developing and analyzing complex SQL across various RDBMS platforms (Microsoft SQL Server, Oracle).</li><li>Expert-level knowledge of data modeling, with a strong understanding of different data structures and their optimal use cases.</li><li>Experience creating scripts with Python</li><li>Hands-on experience with ETL tools (Informatica).</li><li>Proficiency in creating high-quality entity-relationship diagrams (ERDs).</li><li>Strong writing skills for developing user and system documentation.</li><li>Extensive experience with AWS Cloud Data Warehousing technologies.</li><li>Proven ability to build and support data warehouse solutions using core AWS services (S3, DMS, Glue, Lambda), adhering to AWS architectural best practices.</li><li>Development and modeling expertise with Amazon Redshift.</li><li>Experience with AWS service APIs, AWS CLI, and SDKs for application development.</li><li>Proficient in developing, deploying, and debugging cloud-based applications on AWS.</li><li>Skilled in utilizing CI/CD pipelines for application deployment on AWS (GitLab, Terraform, DBMaestro).</li><li>Strong understanding of cloud-native applications, with the ability to write code aligned with best practices.</li><li>Experience in writing code for serverless applications.</li><li>Knowledge of AWS security best practices (e.g., using IAM roles instead of embedding access keys in code).</li><li>Experience with visualization tools (Tableau)</li><li>Experience working on an Agile team</li><li>Understanding of application lifecycle management</li><li>Understanding of the use of containers in the development process</li><li>Ability to author, maintain, and debug code modules on AWS.</li></ul>

Data Engineer Jobs in San Mateo, CA

Lead Data Engineer
January 2, 2025

Senior Data Engineer
January 2, 2025

Data Engineer
January 2, 2025

Data Engineer
January 2, 2025

Data Engineer Databricks
January 2, 2025

Senior Data Engineer
January 2, 2025

Data Engineer
January 2, 2025

Sr Data Engineer
January 2, 2025

Cloud Data Engineer
January 2, 2025

Quick filters

Top companies

Data Engineer Jobs in San Mateo, CA

Lead Data EngineerJanuary 2, 2025

Senior Data EngineerJanuary 2, 2025

Data EngineerJanuary 2, 2025

Data EngineerJanuary 2, 2025

Data Engineer DatabricksJanuary 2, 2025

Senior Data EngineerJanuary 2, 2025

Data EngineerJanuary 2, 2025

Sr Data EngineerJanuary 2, 2025

Cloud Data EngineerJanuary 2, 2025

Quick filters

Top companies

Lead Data Engineer
January 2, 2025

Senior Data Engineer
January 2, 2025

Data Engineer
January 2, 2025

Data Engineer
January 2, 2025

Data Engineer Databricks
January 2, 2025

Senior Data Engineer
January 2, 2025

Data Engineer
January 2, 2025

Sr Data Engineer
January 2, 2025

Cloud Data Engineer
January 2, 2025