Who we are?
ITSG is an innovative software house company operating on the Polish market for over 15 years. We stand out due to the complexity and innovativeness of the completed projects and implemented products for companies from the financial institutions, medicine, pharmacy, telecommunications, and media sectors.
We are looking for a Senior Data Engineer for the Warsaw Data Science Lab, to support us in a project for a global leader in the financial transactions industry. . Who will be responsible for designing, developing, and implementing Data integration and Analytics solutions on SQL Server and/or Hadoop on premise. This position requires close collaboration with business partners to understand Business goals, solicit requirements, and design, develop, and implement Data Analytics solutions.collaboration with business partners to understand Business goals, solicit requirements, and design, develop, and implement Data Analytics solutions.
Duties and responsibilities:
- Build resilient data pipelines for reporting and data science/ML use cases
- Develop workflows and ETL/ELT code using Hive scripts, Spark, Airflow, NiFi, Sqoop, Oozie, and other utilities on Hadoop.
- Participate in technology project delivery activities such as gathering Business requirements, conceptual approach, design, development, test case preparation, unit/integration test execution, and support documentation.
- Develop the source to target mapping documents and ETL/ELT code to load data from the source to the Data warehouse or target application systems
- Partner with IT groups such as CIT, Engineering, Product, Security, and Infrastructure on project delivery activities and security remediations
- Provide insights from data and present them to IT and non-technical users to improve operations and productivity
- Unit test data loads and write scripts for data validation
- Support QA, UAT, and performance testing phases of the development cycle and implement DevOps principles from development to deployment to production
- Provide Production support for jobs running on the cluster and monitor and manage SLA for downstream applications.
- Build data quality frameworks
- Build analytical applications using the Hadoop ecosystem using packaged or open-source technologies
Requirements:
- Minimum 5 years of experience in developing and delivering Analytics projects in the field of ELT/ETL and reporting.
- Completed major studies (Computer Science, Software Engineering or related)
- Experience in data management in SQL Server
- Experience in writing complex SQL queries
- Experience in integrating with internally developed APIs
- Experience in Hadoop development and data integration in Hadoop, Hive, Spark, Presto, Airflow, Sqoop, NiFi etc preferred.
- Experience in building data pipelines for machine learning/advanced model support analytics
- Experience in building data engineering pipeline frameworks
- English at a communicative level
Nice to have:
- Experience in system design and infrastructure construction and maintenance
- Experience in scripting Python, Shell/Perl, scheduling and version control, CI/CD technologies
- Experience with real-time data processing assimilation using Kafka/Spark Streaming
- Strong skills in identifying, analyzing, and solving problems
- Ability to learn new information technologies and apply them quickly
- Exceptional communication and customer engagement skills, able to interact effectively with diverse groups of global stakeholders, both technical and business users
- Experience working in a globally dispersed team
We offer:
- Salary: 130 - 170 PLN+vat /hour
- Hybrid work 2 - 3 set days a week work from the office in Warsaw
- Work with an experienced team of experts using the latest tools and technologies
- Individual training and career growth opportunities
- Benefits: English lessons, Multisport cards, private medical insurance, integration events
