Databricks Experts in Frankfurt

• Implementation of specified DevOps solutions to automate infrastructure (Terraform, Bicep, CloudFormation, Ansible) on-premises datacenter (Ovirt, Proxmox, Ceph Cluster, MinIO) and private cloud.
• Administration, configuration and implementation of CI/CD DevOps pipelines (GitLab, GitFlow) to support development process (Artifactory, Prometheus, Istio, service mesh, Helm Chart, OpenShift (Red Hat Enterprise) / Kubernetes cluster), Red Hat Satellite.
• Administration, setup, monitoring and patching of Linux infrastructure based on Red Hat Enterprise for Dev, Test and QA.
• Use of Scrum and Kanban methods.
• Administration, configuration and implementation of security standards for deploying on Dev, Test, QA and Prod stages of the new ePA applications.
• Development of new plugins and add-ons needed on current infrastructure.
• Database support.
• Data analytics support (Python, Spark, Pandas, Power BI, Splunk Enterprise).
• Implementation of best practices for DevSecOps and BizDevOps using GitOps (ArgoCD), Streamlit framework, Semaphore Ansible UI.
• Configuration and testing of iperf, uperf, sysbench using benchmark-operator for external source data and IoT/MDM devices, creating reports via ELK / OpenSearch.
• Building a new Databricks platform to collect and analyze big data from different sources and IoT devices into Hadoop framework (Python, Pandas, PySpark, Power BI, Apache Airflow).
• Building backend data aggregation and processing to automate configuration deployment between different OpenShift clusters and big data framework (Python, Pandas, PySpark, Apache Spark, PostgreSQL, Django 2, Ansible Automation, Jira JSM).
• Building a new ML pipeline platform using Kubeflow, TensorFlow, KServe.
• Data extraction, transformation and loading from different data sources including structured and unstructured data to analytic DWH / big data cluster using Python, Pandas, Polars, Power BI, Django backend and PostgreSQL.
• Setup of new DevOps Test and QA HashiCorp Vault cluster for PKI and IAM.
• Configuration and testing of automated patching based on CVSS score, SIEM-integrated CVEs.
• Use of Nexpose and InsightVM to scan vulnerability events in network, host, container and application.
• Design and implementation of secure and scalable AWS architectures including VPC, EC2, S3, RDS and Route53 and similar setups on Azure and GCP.
• Automated system provisioning and deployment using CloudFormation templates.
• Configuration of IAM roles, policies and permissions to ensure secure access control.
• Patch management, backup automation and disaster recovery setup on AWS infrastructure.
• Monitoring and optimization of system performance using AWS CloudWatch and AWS Trusted Advisor.
• Support of VMware services (vSphere, Aria, Horizon) and the virtual desktop environment.
• Development and maintenance of CI/CD pipelines using Jenkins, GitLab CI/CD and AWS CodePipeline with interface to Nutanix.
• Configuration of AWS CloudWatch to monitor application performance and system events.
• Planning and execution of migration of on-premises applications to AWS cloud platforms.
• Deployment of containerized applications using Docker and Kubernetes in AWS environments.
• Deployment of internal software packages between availability zones using AWS CodeDeploy.
• Building and deploying ML models using Scikit-learn, XGBoost and Spark MLlib including hyperparameter tuning, model evaluation and production deployment.

• Datastage 11.7, dbt, Oracle 19, Python 3.12 / PySpark 3.5, Azure GitHub, Azure DevOps, Automic
• Development of migration jobs to transfer data from the collection DWH to the new Risk Mart, as well as development of ETL pipelines to migrate historical data from the old Mart to the new Risk Mart.
• Storage of the silver layer on Hadoop and the gold layer in Oracle.
• Translation of DataStage jobs into dbt to publish reporting data in Google Cloud to a PostgreSQL database.
• Creation and optimization of complex SQL queries for data extraction from a data vault, taking into account historical data in the point-in-time tables.
• Creation of Oracle table definitions (DDL) and adjustment of existing stored procedures.
• Versioning changes in GitHub and deployment via the CI/CD portal.
• Refactoring long-running DataStage jobs into Python using PySpark to reduce server load.
• Migration of SAS scripts to PL/SQL, including new development of distribution functions that have no direct equivalent in Oracle.
• Development of Automic jobs to run DataStage pipelines and Python scripts (PySpark jobs) that control the population of the SME and institutional risk tables in the Risk Mart and perform business calculations.
• Participation in the agile process, including creating user stories, estimations, and planning in Azure DevOps.
• Handling Azure DevOps tickets and close collaboration with testers and business teams for error analysis and resolution.

• Advising on the use of analytics and BI tools and services in the Microsoft Azure stack (e.g. MS Fabric, Synapse Workspaces and dedicated SQL pools, SQL Database, PostgreSQL, Snowflake, Databricks, Data Factory, SSIS, Analysis Services, Function Apps, Power BI, ML)
• Independently designing analytics solutions with Python, SQL, etc.
• Designing and implementing ETLs and data pipelines
• Creating and maintaining APIs
• Independently applying CI/CD, testing, and version control
• Data modeling
• Model development and optimization
• Anomaly detection with AI
• Predictive analytics

Used technologies:

• Snowflake
• Fabric
• Azure Synapse Analytics
• Azure DataFactory
• Azure Data Lake
• Azure DevOps
• Databricks
• Spark
• CI/CD
• SQL Database
• Python
• Power Platform

• Provided leadership as Product Owner for the DMS module in an agile multi-team setup
• Coordinated the nationwide rollout including requirements management and roadmap development
• Built and maintained the product backlog focusing on scalability, usability and data protection
• Conducted sprint reviews, refinements and stakeholder demos with over 15 involved teams
• Developed a rollout and training concept for successful launch at over 200 government agencies
• Introduced a reporting and monitoring dashboard to measure usage success
• Worked closely with architects, QA and operations to ensure integration into the existing system landscape

• Developed a retrieval-augmented generation (RAG) application using LangChain and LangGraph for corporate document parsing, delivered as an installable Electron desktop application with local AI models via Ollama.
• Currently providing ongoing AI feature support for the Loan Pricing Tool at Hamburg Commercial Bank, with a commitment of three days per month.
• Architected Kubernetes-native solutions, including Helm chart configuration and Azure DevOps pipeline integration.

• Evaluation of potential providers (Snowflake vs Databricks) and design of the analytics data platform using Databricks
• Data storage and ingestion layer with Amazon S3
• Creation of ETL processes and data transformations with AWS Glue and Databricks Notebooks
• Orchestration with AWS Glue Workflow, Databricks Workflow and Databricks DLT
• Processing of unstructured data including text, image and video
• Databricks workspace setup and administration
• Setting up a medallion architecture to ensure data quality
• Evaluation of possible BI tools (Power BI, AWS QuickSight, Tableau)
• Establishing MLOps using MLflow
• Introducing data governance and data lineage using Unity Catalog

• Ensuring stable operation of business-critical applications in the energy environment (KRITIS-adjacent systems), coordinating with IT operations on certificates, permissions, and security-relevant logging
• Continuous monitoring of interfaces, processes, and system states including analysis of deviations and performance issues
• Development and setup of secure interfaces including authentication and access concepts
• Incident management: prioritization, root cause analysis, coordination of issue resolution with IT operations, business units, and external service providers
• Change and release management including coordination, test coordination, go-live, and post-live support
• Product owner for operational systems in day-to-day operations (forecasts, schedule management, market data import and export, contract management, regulatory reporting)
• Documentation of operational processes, changes, and incidents to ensure traceability and auditability
• Automation and digitization of operational processes
• Introduction and use of AI-supported analysis and monitoring approaches and agents with Microsoft 365 Copilot

• Evaluated search results for relevance, accuracy, and quality based on given guidelines.
• Conducted data annotation and content labeling for AI training models.
• Assessed user intent to refine and enhance search engine algorithms.
• Provided linguistic insights for multilingual search optimization.
• Reviewed AI-generated responses to improve natural language processing (NLP).

• Design, architecture & development/programming of ETL/ELT data pipelines, DWH, BI solution
• Technical project lead, POC – proof-of-concept creation
• Liaison between business units and technical teams
• Azure DevOps Boards & Jira
• Data modeling & data engineering – data warehouse & data mart
• Azure (Data Factory, Azure SQL, Azure DevOps CI/CD, Azure Data Lake V2, Business Central REST API, OData API, OAuth2 tokens)
• SharePoint lists & API for ADF, Firebird DB, Postgres DB, DB2
• Power BI (Power Query), DAX, Excel PBI add-on, GIS data
• Automated ETL process monitoring/logging, performance monitoring, error monitoring – capturing & resolution
• Index performance tuning & statistics monitoring, Transact-SQL
• Data security – MFA (multi-factor authentication) & OAuth2, MS Graph, Azure networks & firewalls, gateways, roles, user groups – with read/write permissions
• Sources – Vario Bill, Camunda, Radius, Geo Database, OTRS, PAST, MS Dynamics Business Central, Azure Blob Data Lake, SharePoint lists

• Global marketing analytics for Hitachi Energy as part of a global data modernization initiative aiming to enhance data retention, historical data availability and provide Eloqua's 2-year retention for remote interaction reporting and analytics.

• Analyzed Eloqua's default retention policy and identified risk of data loss for records older than two years.

• Designed and implemented historical data preservation strategy by creating transformed tables in the target data platform to archive older data while ensuring data quality dashboards.

• Collaborated with the Power BI team to re-point dashboards from raw Eloqua imports to the newly created archival layer.

• Leveraged Jira to track and manage data engineering tasks, bugs, and feature requests across Agile sprints; coordinated backlog prioritization and task assignment to align data pipeline development with business needs.

• Power BI dashboard optimization:

• Worked closely with business stakeholders to assess and understand reporting needs for reverse customer data.

• Designed and implemented incremental refresh in Power BI to ensure daily updates without full data reloads.

• Collaborated with Azure data engineers to optimize data processing and publication pipelines.

• Stakeholder communication & data modeling:

• Acted as liaison between Group Data Office and Technology Office to align data modelling standards.

• Gathered requirements from data engineering team and participated in weekly status meetings to provide implementation updates and resolve blockers across teams in Germany, Poland, and India.

• Documentation & quality assurance:

• Prepared end-to-end technical design documentation, data flow diagrams, and Power BI audit guides for future reference.

• Participated in UAT sessions with business users to validate data outputs and report accuracy.

• Engineered and automated a data pipeline using GitLab CI/CD for data ingestion, validation, and loading into a central database.
• Developed Python scripts for data validation and transformation, ensuring data quality and compliance with metadata standards.
• Managed the entire data lifecycle from file-based repositories to a structured SQL Server database.
• Worked in an interdisciplinary team to establish a central database for-omics data and ensure reproducibility of computational analyses.

• Designed and implemented the agent-based AI architecture for a company-wide platform to securely deploy LLM-based agents
• Designed and implemented end-to-end RAG pipelines from multiple sources: document preprocessing, chunking strategies for different document types, embeddings, retrieval with re-ranking, and robust prompt orchestration
• Developed a modular context engineering framework with skill architecture, context isolation, and dynamic resource management; human-in-the-loop control for enterprise tool integrations
• Built the CI/CD pipeline, testing strategy, tracing on the software side as well as automated LLM and agent evaluations, red team testing and tracing, and handed over to a reproducible production environment (ISO27001 and SOC2 compliant)