Jorge Machado-Data Expert

I designed and developed an AI-powered multi-tenant platform on Azure that transforms SAP process recordings into technical documentation, presentations and automated tests, processing over 15,000 process recordings for enterprise customers like Würth. I owned the architecture, the production releases and the DevOps setup. I also designed a multi-tenant system with SSO and role-based access on Azure. Implemented an MCP Server with Dynamic OAuth Authentication.

Main Tasks:

• Sprint planning and feature preparation
• Design the multi-tenant platform architecture (FastAPI, SQLAlchemy, PostgreSQL row-level security for tenant isolation)
• Develop AI pipelines with Prefect for transcription (Azure Speech API), document generation and SAP screen-recording analysis (Claude, gpt-4-mini)
• Design and implement an MCP server to expose tenant knowledge to LLM clients (Claude), with async retrieval and reranking
• Implement LLM cost tracking, rate limiting and client pooling for Anthropic/OpenAI/Azure OpenAI endpoints
• Set up CI/CD: Docker images to Azure Container Registry, GitHub Actions, Azure Static Web Apps, Alembic migrations in containers
• Manage production releases and execute live data migrations for enterprise customers
• Define engineering standards and architecture patterns for the team

Environment: Azure / Azure Foundry / Python / FastAPI / Prefect / React / PostgreSQL

I designed and developed the migration from SAP HANA to Databricks. I designed and implemented multiple ETL pipelines on Databricks with incremental loading. With the migration, we saved over €50,000 per year in SAP costs. Databricks on Azure

Main Tasks:

• Design the Datalake architecture for Databricks in medallion architecture
• Develop a pipeline to deploy code from dev to prod environments
• Write ETL extraction with PySpark in incremental mode for extracting SAP data into Databricks
• Deploy DBT to create the dimension and fact tables
• Help juniors get up and running with the new platform
• Set up permissions and concepts
• Integrate streaming data from Kafka
• Set up AI use cases like FP-Growth on custom data
• Time series prediction

Environment: Databricks / Azure

In this project I designed and implemented multiple Spark and streaming pipelines using multiple services from AWS. Mainly gathering information in the travel / trains domain (delays, departures, etc). The main use case was delta processing from small files into a multi-terabyte datalake.

Main Tasks:

• Develop streaming applications using AWS Kinesis and Lambda functions
• Develop Apache Spark data processing applications using AWS Glue, AWS S3 and Postgres DB
• Develop KPIs for different applications including time travel analytics
• Several prototypes for Spark Streaming on AWS Glue
• Use advanced features from Apache Hudi
• Business processing monitoring and metrics via AWS CloudWatch
• Infrastructure as code using AWS CDK
• Roll out DBT tools for developers and CI/CD pipelines on GitLab
• Evaluation of Databricks, AWS Athena and Snowflake
• Lead developer helping other developers speed up development cycles
• IoT 4.0 pipelines

Environment: AWS

Develop a data warehouse for the Concept Office ERP system.

Main Tasks:

• Developing dbt tool workflows and star schemas for the data warehouse
• Developing ELT workflows with Stitch Data
• Developing dashboards with Power BI in Azure Cloud

Environment: Google BigQuery and DBT

This project was focused on designing and implementing multiple Apache NiFi clusters over a DMZ and a datacenter.

I developed custom processors in a very short time, and the OPC UA protocol was used to extract data from OPC UA servers.

Main Tasks:

• Developing Java 11 custom NiFi processors, including unit tests
• Creating Ansible playbooks for NiFi deployment with SSL
• OAuth authentication + X.509

Environment: On Premise

Working on the AD-Vantage Program, on self-driving car data

Environment: MapR + OpenShift (500+ nodes)

Main Tasks:

• Developing data pipelines using Spark and Airflow for self-driving cars
• Generating metrics for geospatial applications
• Ingesting data into Elasticsearch using Apache Spark
• Functional programming with Scala

Technologies Used:

• MapR cluster (Hadoop), OpenShift
• Elasticsearch + Kibana
• Apache Airflow
• Kafka Streams

In this project we are ingesting huge amounts of data via Kafka into Accumulo.

The whole Hadoop environment is Kerberized.

Main Tasks:

• Writing Kafka connectors to ingest data
• Kerberizing applications for Hadoop / Kafka / Kafka Connect
• Creating statistics plans for RDF4J queries over Accumulo
• Creating Apache NiFi workflows
• Introducing Git flow automation, continuous integration, and Docker automation
• Kafka Connect setup with Kerberos on Google Kubernetes
• Writing Java applications based on RDF (web semantics)

Environment: Cloudera Hadoop

In this project, the main goal is to integrate Spark more deeply into HBase and architect a new alerting and computing framework based on Spark Streaming. Every deployment is based on Docker.

Main Tasks were:

• Creating reports in Spark jobs over history data
• Custom Spark data sources for HBase and aggregation for data exploration.

Technologies Used:

• Apache HBase with Phoenix JDBC
• Apache Ambari / Hortonworks
• Apache Spark
• Scala and Java
• Vert.x Server
• Docker

Working with R&D on data from cars to perform TensorFlow GPU trainings

Environment: Multiple Mapr clusters (30+ nodes), NVIDIA GPUS (Tesla), Apache Mesos

Main Tasks:

• Developing Data pipelines using Airflow and Apache Spark
• Developing end to end monitoring based on Prometheus
• Developing real time data pipelines based on docker, Kafka and Python.
• Deploying Apache Marathon with Mesos and GPUS
• Architecture for Migration from Mesos to Kubernetes
• Jenkins pipelines for building Docker images to be used Mesos on GPU clusters
• Several Infrastructure tasks done on ansible for High Availability