PySpark Experts in Germany

Project: Defect Generation on Test-Bench Images of Metal Surfaces Environment: Automated Visual Inspection (AVI), Metallurgy & Manufacturing

• Objective & Implementation: Designed, architected, and trained Generative Adversarial Networks (Pix2PixHD / SPADE) for image-to-image transformation. Targeted generation of synthetic material defects (e.g., cracks, inclusions, scale) on rough metal surfaces under real test-bench lighting conditions for privacy-compliant and efficient dataset expansion (data augmentation).
• Technical Design: Implemented robust Generative AI and computer vision pipelines in Python and PyTorch. Used semantic segmentation approaches for mask-controlled defect synthesis and subsequent evaluation with EfficientDet object detection models.
• Business Impact: Massive dataset upscaling (10x) without time-consuming and costly physical test-bench runs, while significantly improving the detection performance of automated inspection systems.

Technologies & Skills Used: Python | PyTorch | SPADE | Pix2PixHD | EfficientDet | Machine Learning | Semantic Segmentation | Computer Vision

• Development of a production-ready Enterprise AI solution for the automated processing of invoices and business documents
• Integration of Azure AI Document Intelligence and LLM technologies into existing business processes
• Development of robust REST APIs for automated document processing and system integration
• Extraction, validation, and storage of structured invoice data in Azure SQL as a base for analytics and machine learning models
• Development of an AI-based recommendation engine with machine learning and deep learning to generate personalized product recommendations based on historical purchase data
• Implementation of logging, monitoring, error handling, and validation mechanisms for stable production use
• Collaboration with business teams to define business rules and integrate the solution into existing enterprise processes

Technologies: Python, Azure AI Document Intelligence, Azure OpenAI, Azure SQL Database, REST APIs, Machine Learning, Deep Learning, OCR, Pandas, JSON, Workflow Automation

• Development of an AI-powered project organization tool for a civil engineering company that intelligently links project, task, tender, schedule, and document data through a knowledge graph.
• Implementation of AI features for document analysis, information extraction, context-based assistance, and voice-based data capture based on Microsoft Azure AI, reducing administrative effort, making information available faster, and supporting project teams in decision-making.
• Tech stack: Python, React, TypeScript, FastAPI, Claude Code, Codex, Graphify, PostgreSQL, Microsoft Azure AI Foundry, Azure OpenAI, Azure AI Speech, Azure AI Document Intelligence, Microsoft Graph, Microsoft Entra ID, Docker, Git, CI/CD.

• This project involved extraction of intelligence data to support report writing for a company that provides geopolitical, global, commercial intelligence. The data have been gathered from a number of resources (interview transcripts, online data, internal documents), and then a knowledge base has been build from it. This was the basis of a complex RAG system, that was evaluated against a golden dataset. Agents have been used to find out the contradicting intelligence, the statements supporting each other, and to store back the generated knowledge.

Used: Python, RAG, LangGraph, LangChain, deepeval, MCP

• Led enterprise reporting modernization by migrating legacy SSRS reporting solutions to Power BI, supporting 500+ business users while ensuring full GDPR/DSGVO compliance.
• Designed and optimized Power BI and Microsoft Fabric semantic models using star schema, dimensional modeling, advanced DAX, and performance optimization techniques, reducing query latency by 25%.
• Delivered 20+ executive and operational dashboards featuring KPI scorecards, drill-through, bookmarks, and row-level security, improving reporting efficiency by 20%.
• Enabled self-service analytics through governed Power BI datasets, dataflows, and gateway architecture, increasing business-led reporting adoption by 35%.
• Configured an incremental refresh policy and query folding for a 50+ million row sales dataset, reducing daily report refresh times by 85%.
• Deployed automated ETL/ELT pipelines using Azure Data Factory, Microsoft Fabric, and Snowflake, reducing reporting delivery timelines by 40% through workflow automation.
• Spearheaded Microsoft Fabric analytics modernization initiatives including lakehouse architecture, OneLake integration, and centralized data platform development, reducing data latency from 2 hours to 20 minutes.
• Translated business requirements from 15+ stakeholders into scalable Power BI semantic models and dashboards, improving reporting consistency and reducing ad-hoc reporting requests by 25%.
• Applied Microsoft Copilot and generative AI tools to accelerate SQL development, DAX authoring, technical documentation, and testing activities, reducing development effort by approximately 15 hours per week.

Tools: Fabric, AWS, dbt, Power BI, SQL, DWH, R, Python

• Supported customers in implementing an architecture design for extracting and preparing data
• Planned the design and implementation of the BI and DWH platform
• Ensured the scalability and performance of the data platform

• Architected and deployed private cloud data platform on VMware vSphere, integrating Greenplum MPP, Apache Kafka, Kubernetes, and Apache Solr, and developed real-time ingestion pipelines with Kafka Connect and Schema Registry.
• Led Oracle Exadata to Greenplum migration, rearchitected data models, optimized storage, implemented RabbitMQ with Debezium for CDC, and deployed VectorDB for Generative AI.
• Designed and executed multi-cloud migration PoC across AWS, Azure, and GCP, defined KPIs for throughput, latency, and cost efficiency, executed bulk data transfers, validated analytics and streaming workloads, and delivered full-scale architecture recommendations.
• Assessed legacy on-premises infrastructure and designed modern cloud-native data platforms using Greenplum and containerized microservices, advising on scalability, disaster recovery, and high-availability.

• Building, implementing and selling automated Data Science solutions such as Scorecard Factory and Forecast Factory
• Implementation of automated end-to-end cloud processes
• Development of LLM and NLP models
• Creation of interactive reports
• Support for national and international large corporations as well as medium-sized companies in implementing ML projects

Built Python-based AI microservices and integrations for an AEC/VR Unity-based SaaS app, focusing on LLM/VLM capabilities, retrieval-backed systems, RESTful APIs, containerized deployment, and an automation microservice for the CAD-to-Unity pipeline.

• Developed a custom Hybrid A* based algorithm in C# to simulate hospital scenarios and detect early-stage design conflicts from collision/spatial data and generate structured reports.
• Solved and automated the time-consuming problem of converting CAD files to usable Unity environments with a custom-engineered and real-time pipeline using a ZeroMQ-based communication layer to distribute workloads across multiple processes and achieve real-time performance.
• Built a Dockerized FastAPI pipeline for CAD-to-Unity automation, combining vision-based object matching, image embeddings, and precomputed metadata to automatically map CAD objects to Unity behavior scripts, assign properties, and reduce repeated AI inference calls.
• Created documentation and examples to help technical users understand, configure, and extend the AI automation pipeline.

Architected and maintained data products for renewable energy operations, covering wind turbine, grid-meter, and weather data. Built scalable ETL/ELT pipelines in Azure Databricks using Delta Lake (bronze/silver/gold layers) and processed data in various formats, including structured and semi-structured data. Contributed to a data quality framework supporting table and column documentation, outlier detection, and completeness metrics across all datasets within a data product. In addition, implemented a DORA KPI Databricks dashboard used across all data products. Optimized CI/CD processes in Azure DevOps to streamline deployment across development, test, and production environments.

Technology stack: Azure Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Azure Data Lake, APIs, Dremio, Azure DevOps, YAML, Git, Databricks Workflows, Application Insights, Terraform, OpenAI API, Codex, LLM-assisted workflows

• Design and build of a fully isolated AWS offline environment with no outbound internet access for running a browser-based business application
• Design and implementation of a proxy and response service that terminates all external application calls inside the VPC and serves them from locally stored content; identification of the actual communication needs through measurement-based DNS query logging
• Creation of architecture designs and decision papers including a comparison of options (Application Load Balancer with Lambda and S3, reverse proxy on EC2, private API Gateway) assessed by operational effort, cost, and availability
• Transfer of the solution and operations documentation previously available only for Azure to an AWS target architecture, including reassignment of all services and operational processes
• Automated rollout as Infrastructure as Code (Terraform, CloudFormation) with CI deployment via GitHub Actions, plus setup of private DNS zones and an internal certificate chain for operation without internet access
• Creation of architecture, deployment, and operations documentation and handover to the customer
• Build-up of a private cloud platform on OpenStack at provider TelemaxX with Terraform, including FortiGate HA clusters, FortiManager, and Kubernetes
• Introduction of Policy as Code (Open Policy Agent, Conftest) as well as development of MCP servers (Model Context Protocol) to connect AI assistants to operations and project tools

Successes:

• Made the business application fully operable without internet access for the first time; the cause of the loading error was narrowed down systematically to missing CORS headers after the likely certificate issue was ruled out
• Fully transferred an existing Azure concept to AWS and replaced the manually created environment with a reproducible, CI-based rollout

Technology stack: AWS (VPC, Application Load Balancer, Lambda, S3, Route 53 private hosted zones and Resolver query logging, IAM, CloudWatch, EC2, CloudFormation), Infrastructure as Code (Terraform, CloudFormation, Remote State), CI/CD (GitHub Actions with OIDC, Azure DevOps Pipelines), OpenStack, FortiGate, FortiManager, Kubernetes, Policy as Code (Open Policy Agent, Conftest), offline and air-gap architectures, PKI & certificates (internal CA, TLS, CRL/OCSP), DNS, network segmentation, Linux, Windows Server, Python, Bash, PowerShell, YAML, JSON, architecture design & decision papers, documentation (Confluence, Markdown), Generative & Agentic AI (Model Context Protocol, Agentic AI Coding Tools)

I designed and developed an AI-powered multi-tenant platform on Azure that transforms SAP process recordings into technical documentation, presentations and automated tests, processing over 15,000 process recordings for enterprise customers like Würth. I owned the architecture, the production releases and the DevOps setup. I also designed a multi-tenant system with SSO and role-based access on Azure. Implemented an MCP Server with Dynamic OAuth Authentication.

Main Tasks:

• Sprint planning and feature preparation
• Design the multi-tenant platform architecture (FastAPI, SQLAlchemy, PostgreSQL row-level security for tenant isolation)
• Develop AI pipelines with Prefect for transcription (Azure Speech API), document generation and SAP screen-recording analysis (Claude, gpt-4-mini)
• Design and implement an MCP server to expose tenant knowledge to LLM clients (Claude), with async retrieval and reranking
• Implement LLM cost tracking, rate limiting and client pooling for Anthropic/OpenAI/Azure OpenAI endpoints
• Set up CI/CD: Docker images to Azure Container Registry, GitHub Actions, Azure Static Web Apps, Alembic migrations in containers
• Manage production releases and execute live data migrations for enterprise customers
• Define engineering standards and architecture patterns for the team

Environment: Azure / Azure Foundry / Python / FastAPI / Prefect / React / PostgreSQL

• Led strategic software development of fusion algorithms for precise object tracking, trajectory prediction, and environment modeling based on multimodal sensor data (e.g., camera, LiDAR, radar, GNSS, IMU)
• Developed and implemented navigation algorithms for autonomous vehicles, including path planning, obstacle avoidance, and sensor fusion of visual, inertial, and distance-based sensor sources
• Automated extraction and training processes with CI/CD
• Developed and optimized data pipelines and processes in Microsoft Azure using Apache Spark, Databricks, and PySpark
• Developed and optimized embedded software for automotive control units
• Designed latency-critical software for real-time control in robotic systems with RTOS (freeRTOS, SAFERTOS)
• Used the Vector toolchain (CANdela, DaVinci, CANoe) for configuration and diagnostics
• Optimized existing data pipelines and processes (ETL, data warehouse, SQL)
• Developed and trained machine learning models using PyTorch
• Created deep-learning-based object detection and visual SLAM algorithms, trained on combined data from camera, LiDAR, and IMU sensors
• Implemented computer vision algorithms for object detection and classification in robotic systems using OpenCV and YOLO, utilizing synchronized image and depth data
• Implemented behavior-based control systems for autonomous robots using ROS2 Behavior Trees
• Performed testing, release, and integration of sensor fusion algorithms into automotive production programs
• Ensured adherence to proper software development processes and safety standards to guarantee high data quality (MISRA, ISO 26262, ASPICE)

• Designed and deployed enterprise Retrieval-Augmented Generation (RAG) applications using LangChain, LangGraph, vector databases, embeddings, and open-source LLMs served through vLLM on GCP GPU infrastructure.
• Built agentic AI workflows using LangGraph with planning, reasoning, tool execution, persistent memory, session management, and Human-in-the-Loop approval mechanisms.
• Developed LLM-powered automation systems integrating BigQuery, SQL pipelines, and external advertising APIs including Meta, TikTok, Amazon, Snapchat, Google, and Pinterest, reducing manual operational workflows.
• Architected multi-agent AI systems for enterprise analytics and decision-support workflows, enabling autonomous task execution and intelligent data interactions.
• Implemented retrieval optimization strategies including multi-retriever architectures, semantic search, context optimization, and query improvement techniques, improving response relevance by approximately 40%.
• Engineered structured prompting strategies, function-calling schemas, and validation workflows to improve reliability of multi-step LLM applications.
• Designed scalable AI services using Python, FastAPI, Cloud Run, Pub/Sub, BigQuery, Docker, and cloud-native deployment architectures.