Senior Data Engineer (Blockchain, Remote)

Job summary

This role involves developing and optimizing data processing systems for a leading organization in the blockchain and cryptocurrency sector. The position focuses on building scalable data pipelines and enhancing data architecture to support high-volume data ingestion and processing.

Qualifications

• Proven experience in designing and implementing cloud-native, distributed data processing systems, preferably on AWS.
• Strong proficiency in Python and Apache Spark, with a solid understanding of performance tuning and distributed computing.
• Experience in backend systems and large-scale data architecture.
• Familiarity with mentoring engineers and leading cross-team initiatives is advantageous.
• Interest in cryptocurrency technology and its data challenges is a plus.

Responsibilities

• Construct cloud-native data ingestion and aggregation processes handling large volumes of data daily.
• Develop and optimize Spark jobs in Python for detecting market manipulation and fraud.
• Build and enhance batch applications and real-time streaming pipelines processing billions of records.
• Architect and maintain scalable data lakehouse environments using formats like Parquet and Delta Lake.
• Collaborate on scalable API services on AWS to manage high request volumes.
• Modernize the data stack to improve operational capacity significantly.
• Troubleshoot data quality issues and performance bottlenecks in distributed systems.

Skills

• Expertise in Python, Apache Spark, and distributed data processing.
• Familiarity with Terraform and Kubernetes for orchestrating data workloads is beneficial.
• Experience with modern data stack technologies is desirable.

Education

• Relevant technical background in data engineering or a related field is preferred.

Tools

• Technologies used include Python, Java, Spark, Flink, Databricks, AWS, Kubernetes, Terraform, GitHub, PostgreSQL, and Snowflake.