Job Description:
Data Engineer is required to be proactive in designing, building and maintaining data systems. The role also requires cross-functional collaborations and ensure highest standards of data quality and performance by extending his expertise in data engineering, data architecture, pipeline creation and big data technologies.
Key Responsibilities:
- Understand the values and vision of the organization.
- Protect the Intellectual Property.
- Adhere to all the policies and procedures.
- Design, develop, and maintain scalable data pipelines for data ingestion, processing and storage.
- Build and optimize data architectures and data models for efficient data storage and retrieval.
- Develop ETL processes to transform and load data from various sources into data warehouses and data lakes.
- Ensure data integrity, quality, and security across all data systems.
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
- Monitor and troubleshoot data pipelines and workflows to ensure high availability and performance.
- Document data processes, architectures, and data flow diagrams.
- Implement and maintain data integration solutions using industry-standard tools and technologies (e.g., Apache Spark, Kafka, Airflow).
Required Skills & Experience:
- Expertise on Data Integration, processing & Storage.
- Expertise on Data optimization architecture, data process and data flow.
- Knowledge of Data integration tools like Apache Spark, Kafka & Airflow.
- Proficiency in SQL and at least one programming language (e.g., Python, Scala).
- Experience with cloud data platforms (e.g., AWS, Azure, GCP) and their data services.
- Experience with data visualization tools (e.g., Tableau, Power BI).