Role - Data Engineer
Location - Dallas, TX (Onsite)
Skills:
Must Have:
BS/BA degree or equivalent experience
General: Strong organizational, problem-solving, and critical thinking skills; Strong documentation skills
Cluster Computing frameworks: Proficiency in Spark and Bigdata .

We are seeking an experienced AWS Glue Specialist to join our team and lead the migration of ETL processes from IBM DataStage to AWS Glue. This role requires hands-on expertise in designing, developing, and optimizing ETL pipelines, with a focus on migrating data from various sources, both on-premises and cloud-based. The ideal candidate will have a proven ability to work independently and collaborate with multi-domain teams to ensure successful data integration and transformation.
Key Responsibilities
Lead the migration of ETL processes from IBM DataStage to AWS Glue, ensuring efficiency and reliability.
Design, develop, and optimize ETL pipelines using AWS Glue, incorporating best practices for performance, scalability, and cost-efficiency.
Analyze existing DataStage ETL workflows to identify and implement optimized solutions in the AWS environment.
Extract data from diverse sources (on-premises and cloud) and transform it into a cloud-compatible format using AWS Glue.
Collaborate with cross-functional teams to understand data requirements and ensure seamless data integration across multiple domains.
Independently troubleshoot and resolve technical challenges, ensuring data integrity and compliance with project timelines.
Document technical specifications, data flows, and transformation logic for ETL processes.
Monitor and maintain ETL pipelines to ensure continuous operation, making enhancements as needed.
Support the team with best practices in cloud data management and AWS Glue usage.
Qualifications
5+ years of experience in ETL development, with a focus on data migration.
3+ years of hands-on experience with AWS Glue, including job creation, workflow orchestration, and optimization.
Proven expertise in IBM DataStage and experience in migrating ETL jobs to cloud platforms, particularly AWS.
Strong knowledge of AWS services (S3, Lambda, CloudWatch, Redshift, RDS, etc.) and how they integrate with AWS Glue.
Experience with Python or PySpark for ETL scripting and transformations.
Familiarity with SQL and relational databases, as well as NoSQL databases.
Strong problem-solving skills with the ability to work independently and collaborate across multiple teams.
Excellent communication skills for interacting with stakeholders and documenting processes.
Preferred Skills
Experience with AWS Data Migration Services (DMS) or other data migration tools.
Knowledge of data warehousing concepts and best practices.
AWS certifications (e.g., AWS Certified Data Analytics, AWS Certified Solutions Architect) are a plus.
Familiarity with Agile methodologies and experience working in a fast-paced environment.

**W2 Only**
Job Title:  Lead Data Engineer
Location :  Currently in Nashville, TN metro ( Locals Only )
Working Requirements:  4 days in-office hybrid
Experience & Skills :
3 - 5+ years of Data Engineer experience including API and Database Knowledge
Strong object-oriented programming and design skills
Knowledge of modern database technologies and experience developing database applications on AWS
AWS Serverless, Cloud Security, DevOps, Cloud Migration, Containers
Languages include SQL/NOSQL ( Redshift and Aurora Postgress ); Python, ,  Node.JS , Chart JS, Event Bridge, Step Functions, Lambda, Glue, CloudWatch, CloudFront, AWS DMS
Knowledge in modern data engineering languages such as Python or PySpark, and coding best practices such as CI/CD a plus
Knowledge of the software development lifecycle and concepts such as Agile, SAFE, scrum, CI/CD, and DevOps a plus

Job Summary: We are seeking a highly skilled Python Data Engineer responsible for designing and developing robust data solutions. This role will involve building data load pipelines using Python, Kafka, and Snowflake, ensuring non-functional requirements are met. The ideal candidate will work closely with cross-functional teams to meet both business and IT needs while adhering to industry best practices. Key Responsibilities: Design and develop data load solutions using Python, Kafka, and other data integration methodologies. Implement data loads using Snowflake SQLs and stored procedures. Collaborate with business partners and product analysts to translate business needs into technical solutions. Develop, test, and deploy high-quality, scalable data models and components. Ensure non-functional requirements, such as security, scalability, and reliability, are accounted for during development. Work closely with architects and designers to create optimal data frameworks. Maintain coding standards, perform unit testing, and resolve platform production issues. Identify and troubleshoot environmental issues in development and production environments. Required Qualifications: Expertise in Python, Snowflake, and Apache Kafka. 5 years of experience in data warehousing, ETL tools, and data modeling. Strong understanding of the software development lifecycle and Agile methodologies. Experience working with secure data development processes. Analytical and problem-solving mindset. Strong communication and collaboration skills with the ability to work in an onsite-offshore setup. Preferred Qualifications: Experience in resolving platform production issues efficiently. Familiarity with additional data tools and methodologies beyond core ETL tools. Experience designing secure and scalable data models in cloud environments. Ability to adapt quickly to changing environments and requirements. Education: Bachelors Degree

Day-to-Day Responsibilities:

Will be helping a software client out with a migration from on Prem to AWS. Will be utilizing Python, SQL, Pyspark to build out pipelines to connect to AWS, leveraging Databricks as the primary datawarehouse tool. Most of the work is in the Data Engineering capacity with the pipelines, but some cloud engineering is also needed to help configure. Looking for a senior level resource that can do E2E work, req gathering, design, develop, test, and deployment.

Must Haves:
Senior level
AWS
DATABRICKS
PYTHON
PYSPARK

Education:  Bachelors Degree

Job Summary Client is seeking a highly skilled Data Engineer for a 12-month engagement to design and develop big data applications in a GCP environment. The role involves creating data pipelines, managing workflows, and ensuring data quality for large-scale projects within a collaborative and dynamic team environment. Key Responsibilities Design and develop big data applications using open-source technologies. Work with offshore models and manage project outcomes. Develop logical and physical data models for big data platforms. Automate workflows using Apache Airflow and manage data pipelines via Apache Hive, Spark, and Kafka. Maintain and enhance existing systems, participating in on-call support. Mentor junior engineers, lead standups, and conduct design reviews. Prioritize backlogs and manage JIRA activities. Act as the primary contact for assigned business domains. Required Qualifications Bachelors Degree in Computer Science or related field. 5 years of experience in data warehouse solutions and data products development. 5 years of experience with distributed data platforms (Hadoop, Hive, Spark, Airflow). 2 years of experience with GCP (Dataproc, GCS, BigQuery). Proficiency in Python, Java, or Scala and experience with scripting languages (Perl, Shell). Strong experience with test-driven development, Agile methodologies, and working with large datasets. Preferred Qualifications Experience with Gitflow and Atlassian tools (BitBucket, JIRA, Confluence). Familiarity with CI tools (Bamboo, Jenkins, TFS). Experience with GitHub for source control. Certifications Relevant GCP certifications are preferred. Education: Bachelors Degree Certification: Google Associate certification

Job Summary: We are seeking a highly skilled Python Data Engineer responsible for designing and developing robust data solutions. This role will involve building data load pipelines using Python, Kafka, and Snowflake, ensuring non-functional requirements are met. The ideal candidate will work closely with cross-functional teams to meet both business and IT needs while adhering to industry best practices. Key Responsibilities: Design and develop data load solutions using Python, Kafka, and other data integration methodologies. Implement data loads using Snowflake SQLs and stored procedures. Collaborate with business partners and product analysts to translate business needs into technical solutions. Develop, test, and deploy high-quality, scalable data models and components. Ensure non-functional requirements, such as security, scalability, and reliability, are accounted for during development. Work closely with architects and designers to create optimal data frameworks. Maintain coding standards, perform unit testing, and resolve platform production issues. Identify and troubleshoot environmental issues in development and production environments. Required Qualifications: Expertise in Python, Snowflake, and Apache Kafka. 5 years of experience in data warehousing, ETL tools, and data modeling. Strong understanding of the software development lifecycle and Agile methodologies. Experience working with secure data development processes. Analytical and problem-solving mindset. Strong communication and collaboration skills with the ability to work in an onsite-offshore setup. Preferred Qualifications: Experience in resolving platform production issues efficiently. Familiarity with additional data tools and methodologies beyond core ETL tools. Experience designing secure and scalable data models in cloud environments. Ability to adapt quickly to changing environments and requirements. Education: Bachelors Degree

Job Summary We are seeking a Data Engineer to support the migration of approximately 300 reports and reworking of SSIS packages into Snowflake, with a completion timeline extending to March 2026. The role involves designing and maintaining data pipelines, developing ETL processes, and collaborating with stakeholders to deliver optimized data solutions. Key Responsibilities Migrate and rework SSIS packages into Snowflake. Design, develop, and maintain scalable data pipelines and ETL processes. Perform data manipulation, transformation, and performance tuning using SQL and T-SQL. Implement data warehousing solutions in Snowflake with an emphasis on optimization and reporting. Develop Python scripts for data processing, including zipping and encrypting locally stored files. Collaborate with stakeholders to gather requirements and create data solutions. Manage code and deployments using GitLab across Dev, Test, and Prod environments. Document data processes to ensure data quality and integrity. Required Qualifications 1.5 years of recent experience with Snowflake. Proficiency in SQL, capable of writing complex queries and stored procedures. Ability to modify and read Python code for data processing tasks. Prior experience in the healthcare domain. Preferred Qualifications Experience with data migrations and SSIS packages. Familiarity with GitLab for version control and deployments. Education: Bachelors Degree

Data Engineer Compunnel Jobs

Data Engineer
December 8, 2024

Data Engineer
December 2, 2024

Lead Data Engineer
November 30, 2024

Python Data Engineer
November 26, 2024

Data Engineer
November 24, 2024

Senior Data Engineer
November 22, 2024

Python Data Engineer
November 21, 2024

Data Engineer
November 21, 2024

Data Engineer
November 18, 2024

Snowflake Data Engineer
November 16, 2024

Quick filters

Top companies

Data Engineer Compunnel Jobs

Data EngineerDecember 8, 2024

Data EngineerDecember 2, 2024

Lead Data EngineerNovember 30, 2024

Python Data EngineerNovember 26, 2024

Data EngineerNovember 24, 2024

Senior Data EngineerNovember 22, 2024

Python Data EngineerNovember 21, 2024

Data EngineerNovember 21, 2024

Data EngineerNovember 18, 2024

Snowflake Data EngineerNovember 16, 2024