Apache Kafka Jobs

Our production data pipeline is already live on Azure, but the data-processing layer needs a careful upgrade. I ingest events through Kafka, land them in an Azure Data Lake, then run real-time transformations in Apache Flink. Everything is orchestrated by Airflow on a Kubernetes cluster, with CI/CD handled through our DevOps toolchain. I need a seasoned data engineer who can dive straight into the Flink jobs, refactor the Python code where necessary, tune state management and checkpointing, and release the changes through our existing Kubernetes-based workflow. You will also validate end-to-end data quality in the lake and leave the deployment scripts cleaner than you found them. Deliverables • Refactored Flink job(s) with improved throughput and lower latency • Updated Airfl...

Backend Developer This person builds the core system and integrations. Job Title Backend Developer – Transport System APIs, GPS Integration, Ticketing Platform Job Description Project Overview We are developing a hybrid bus transport management platform that integrates GPS tracking with a custom passenger services system. The system will initially support 250–300 buses and provide real-time telemetry, digital ticketing, and operational monitoring. The architecture combines an existing white-label GPS tracking platform with a custom backend system to support passenger services and transport operations. Responsibilities • Design and develop backend APIs • Integrate white-label GPS tracking system • Implement real-time bus location ingestion • Develop dig...

My trading algorithm is missing profitable entries because the real-time feed reaches it a few seconds too late. The logic itself works; the bottleneck sits in the data path and the supporting back-end services. I need you to trace where the latency creeps in, redesign or optimise that layer, and hand back a snappy, reliable pipeline that keeps quotes, trades and order events truly live. Current situation • Incoming prices arrive from the provider in real time, yet by the time the algo processes them I can see a noticeable lag on the dashboard. • Once the delay starts it snowballs throughout the session until I restart the services. 2. Code or configuration changes that bring the end-to-end delay under 50 ms. 3. A concise hand-off document (setup steps, dependencies, a...