PySpark Expert

• Designed and developed a production-ready data platform based on Microsoft Fabric for an e-commerce company
• Built a scalable Medallion Architecture (Bronze, Silver, Gold) with Lakehouse and Delta Lake as the foundation for analytics and AI applications
• Developed a metadata-driven ETL platform with PySpark and a generic Incremental Loading Framework (SQL Server → Bronze → Silver → Gold)
• Developed reusable components for Microsoft Fabric Notebooks to standardize data integration, transformation, and validation
• Implemented automated data pipelines including monitoring, logging, error handling, and Delta MERGE strategies for stable production operations
• Built a business layer architecture for reporting, data warehousing, and reusable data products
• Integrated heterogeneous data sources via SQL Server, REST APIs, Amazon SP API, and Google Analytics 4
• Worked closely with business departments to analyze business requirements and translate them into scalable data architectures

Technologies: Microsoft Fabric, PySpark, Python, SQL Server, Delta Lake, Lakehouse, OneLake, Data Factory, Data Warehouse, REST APIs, Amazon SP-API, Google Analytics 4

• Optimized the data architecture (Medallion) to better decouple processing steps and improve transparency and reproducibility

• Ensured the technical quality of data processing in Databricks by introducing schema enforcement, data quality checks, and a structured data architecture

• Pipeline orchestration with Azure Data Factory

• Improved and automated the development and deployment process by integrating Git and GitHub Actions

• Technical leadership of the Data Engineering team (3 employees)

• Led workshops to optimize and stabilize the data platform and development process

• Collected and prioritized new requests and maintained the product backlog

• Technologies: Microsoft Azure (Data Lake, Data Factory), Databricks, Apache Spark (PySpark), Python, SQL, Git, Confluence, Power BI, Power Apps, Dataverse, MS Sharepoint, Mural

• Taught ML, BI, and Big Data to Master's students.

Tech: Python, Power BI, SQL, ML, SnowflakeDB, databricks, MS Fabric, MongoDB, LLM

Project: Defect Generation on Test-Bench Images of Metal Surfaces Environment: Automated Visual Inspection (AVI), Metallurgy & Manufacturing

• Objective & Implementation: Designed, architected, and trained Generative Adversarial Networks (Pix2PixHD / SPADE) for image-to-image transformation. Targeted generation of synthetic material defects (e.g., cracks, inclusions, scale) on rough metal surfaces under real test-bench lighting conditions for privacy-compliant and efficient dataset expansion (data augmentation).
• Technical Design: Implemented robust Generative AI and computer vision pipelines in Python and PyTorch. Used semantic segmentation approaches for mask-controlled defect synthesis and subsequent evaluation with EfficientDet object detection models.
• Business Impact: Massive dataset upscaling (10x) without time-consuming and costly physical test-bench runs, while significantly improving the detection performance of automated inspection systems.

Technologies & Skills Used: Python | PyTorch | SPADE | Pix2PixHD | EfficientDet | Machine Learning | Semantic Segmentation | Computer Vision

Built Python-based AI microservices and integrations for an AEC/VR Unity-based SaaS app, focusing on LLM/VLM capabilities, retrieval-backed systems, RESTful APIs, containerized deployment, and an automation microservice for the CAD-to-Unity pipeline.

• Developed a custom Hybrid A* based algorithm in C# to simulate hospital scenarios, detect early-stage design conflicts using collision/spatial data, and generate structured reports.
• Solved and automated the time-consuming task of converting CAD files into usable Unity environments with a custom real-time pipeline. Used a ZeroMQ-based communication layer to distribute workloads across multiple processes and achieve real-time performance.
• Built a Dockerized FastAPI pipeline for CAD-to-Unity automation, combining vision-based object matching, image embeddings, and precomputed metadata to automatically map CAD objects to Unity behavior scripts, assign properties, and reduce repeated AI inference calls.
• Created documentation and examples to help technical users understand, configure, and extend the AI automation pipeline.

• Architected and deployed private cloud data platform on VMware vSphere, integrating Greenplum MPP, Apache Kafka, Kubernetes, and Apache Solr, and developed real-time ingestion pipelines with Kafka Connect and Schema Registry.
• Led Oracle Exadata to Greenplum migration, rearchitected data models, optimized storage, implemented RabbitMQ with Debezium for CDC, and deployed VectorDB for Generative AI.
• Designed and executed multi-cloud migration PoC across AWS, Azure, and GCP, defined KPIs for throughput, latency, and cost efficiency, executed bulk data transfers, validated analytics and streaming workloads, and delivered full-scale architecture recommendations.
• Assessed legacy on-premises infrastructure and designed modern cloud-native data platforms using Greenplum and containerized microservices, advising on scalability, disaster recovery, and high-availability.

• Development of an AI-powered project organization tool for a civil engineering company that intelligently links project, task, tender, schedule, and document data through a knowledge graph.
• Implementation of AI features for document analysis, information extraction, context-based assistance, and voice-based data capture based on Microsoft Azure AI, reducing administrative effort, making information available faster, and supporting project teams in decision-making.
• Tech stack: Python, React, TypeScript, FastAPI, Claude Code, Codex, Graphify, PostgreSQL, Microsoft Azure AI Foundry, Azure OpenAI, Azure AI Speech, Azure AI Document Intelligence, Microsoft Graph, Microsoft Entra ID, Docker, Git, CI/CD.

Real-time AWS threat detection using LLMs, UEBA, and vector analytics

• Built UEBA engine processing millions of CloudTrail/GuardDuty events to detect behavioural anomalies in real time.
• Integrated Amazon Bedrock LLMs for narrative threat analysis and structured automated incident summarisation.
• Designed vector embedding pipeline (MongoDB Atlas) for entity behaviour comparison and semantic anomaly detection.

Tech: Python, FastAPI, AWS, MongoDB Atlas, Amazon Bedrock, Pydantic, Motor (Async).

• This project involved extraction of intelligence data to support report writing for a company that provides geopolitical, global, commercial intelligence. The data have been gathered from a number of resources (interview transcripts, online data, internal documents), and then a knowledge base has been build from it. This was the basis of a complex RAG system, that was evaluated against a golden dataset. Agents have been used to find out the contradicting intelligence, the statements supporting each other, and to store back the generated knowledge.

Used: Python, RAG, LangGraph, LangChain, deepeval, MCP

• Led end-to-end delivery of ML/AI applications for the QuantistryLab simulation platform, from requirements definition and solution design to development, testing, deployment, and production support, coordinating efforts across back-end, DevOps, and ML teams
• Designed and implemented a compound search system with an open-weights LLM, the Pydantic AI agent framework, and the Model Context Protocol (MCP), exposing it through FastAPI endpoints
• Deployed and configured vLLM-based LLM serving on Kubernetes
• Built CI/CD pipelines and Helm charts to automate deployments across staging and production environments
• Instrumented the search system with OpenTelemetry spans and traces, integrating with the Honeycomb observability platform to monitor component-level latency and functional errors
• Developed a custom evaluation dataset and implemented evaluations for retrieval relevance, answer correctness, tool-call accuracy, and end-to-end task success
• Reduced infrared spectrum prediction time by 83% (from 60 to 10 minutes) by replacing a quantum-mechanical model with a pretrained graph neural network; fine-tuned the model using PyTorch, built and containerized the inference pipeline with Docker, and integrated it into the platform
• Implemented an MLflow model registry with MinIO-backed artifact storage
• Advised sales and science teams on ML/AI applications for prospective client projects, providing technical expertise to shape proposed solutions and prototypes

Technologies: AWS, Hetzner, Kubernetes, Helm, Docker, vLLM, Pydantic AI, MCP, Qwen, Llama 3, MLflow, MinIO, GitHub Actions, Python, FastAPI, OpenTelemetry, Honeycomb, PyTorch, Polars, pytest

• Lakehouse build: Designed and built the member & eligibility domain of an Azure Databricks healthcare lakehouse, ingesting SQL Server, file and REST/SFTP sources (JSON, CSV, XML, Parquet) via Azure Data Factory into ADLS Gen2.
• Medallion pipelines: Transformed data through bronze, silver and gold Delta Lake layers with PySpark and Spark SQL; used Delta Live Tables / Lakeflow with built-in expectations for declarative, quality-checked pipelines.
• Platform modernisation: Delivered a SQL Server-to-cloud platform: ADF → ADLS Gen2 → Databricks → dbt Core models → Snowflake → Power BI, replacing legacy reporting with governed analytics.
• Robust ETL: Built metadata-driven, idempotent full, incremental and watermark-driven loads with schema evolution and failure-state handling, improving reliability and re-runnability.
• Data modelling: Designed Kimball star schemas with SCD Type 1 & 2 dimensions; tuned slow queries and pipelines using execution plans.
• Governance & security: Applied Unity Catalog RBAC, lineage and audit logging, with Azure Key Vault secrets and managed identities, to meet HIPAA and GDPR requirements.
• Leadership & mentoring: Led a 6-person team (2 data engineers, 2 analysts, BI developer, QA); mentored engineers, set delivery standards and coordinated with international stakeholders.
• Delivery & AI-assisted work: Ran CI/CD and work tracking in Azure DevOps; built Power BI semantic models with DAX and RLS; used Genie Code, Claude and Copilot to accelerate PySpark/SQL development, DAX and documentation.

• Led enterprise reporting modernization by migrating legacy SSRS reporting solutions to Power BI, supporting 500+ business users while ensuring full GDPR/DSGVO compliance.
• Designed and optimized Power BI and Microsoft Fabric semantic models using star schema, dimensional modeling, advanced DAX, and performance optimization techniques, reducing query latency by 25%.
• Delivered 20+ executive and operational dashboards featuring KPI scorecards, drill-through, bookmarks, and row-level security, improving reporting efficiency by 20%.
• Enabled self-service analytics through governed Power BI datasets, dataflows, and gateway architecture, increasing business-led reporting adoption by 35%.
• Configured an incremental refresh policy and query folding for a 50+ million row sales dataset, reducing daily report refresh times by 85%.
• Deployed automated ETL/ELT pipelines using Azure Data Factory, Microsoft Fabric, and Snowflake, reducing reporting delivery timelines by 40% through workflow automation.
• Spearheaded Microsoft Fabric analytics modernization initiatives including lakehouse architecture, OneLake integration, and centralized data platform development, reducing data latency from 2 hours to 20 minutes.
• Translated business requirements from 15+ stakeholders into scalable Power BI semantic models and dashboards, improving reporting consistency and reducing ad-hoc reporting requests by 25%.
• Applied Microsoft Copilot and generative AI tools to accelerate SQL development, DAX authoring, technical documentation, and testing activities, reducing development effort by approximately 15 hours per week.

Tools: Fabric, AWS, dbt, Power BI, SQL, DWH, R, Python

• Supported customers in implementing an architecture design for extracting and preparing data
• Planned the design and implementation of the BI and DWH platform
• Ensured the scalability and performance of the data platform

Designed and developed an Azure-based AI and data platform transforming distributed project documents into searchable, structured knowledge for an enterprise AI assistant. Combined incremental document processing, hybrid keyword/vector retrieval, knowledge graph extraction, and conversational AI for Microsoft Teams and Microsoft 365 Copilot. Implemented Python document-processing pipelines for PDF, Word, Outlook and text files; Azure Cosmos DB lifecycle management; Azure AI Search; Azure AI Foundry and Azure OpenAI; identity-aware retrieval with Microsoft Entra ID; project authorization; a TypeScript Microsoft Agents SDK agent; Pulumi infrastructure; resilient scheduling, retry and diagnostic tooling. Technology stack: Python, TypeScript, FastAPI, Pydantic, Azure AI Foundry, Azure OpenAI, Azure AI Search, Azure Cosmos DB, Blob/Table Storage, Container Apps/Jobs, Service Bus, Key Vault, Entra ID, Docker, Pulumi, pytest, Microsoft Teams and Microsoft 365 Copilot.