OpenSearch Transform Job: The Case of the Silent Failure and the Ghost Key
Debugging OpenSearch Transform jobs can feel like searching for a needle in a haystack, especially when the error messages ar...
Feb 13, 2026
// research infrastructure
Derek Weitzel
Distributed Research Infrastructure
Designing and operating production-grade distributed research platforms across national collaborations including OSG, PATh, NRP, and IRIS-HEP.
Focus areas: reliable data movement, scalable throughput computing, and long-lived scientific services engineered for production use.
// current focus
- Federated Kubernetes across research domains
- Reliable high-throughput data movement
- Production-grade distributed research platforms
// systems
Projects
OSG A national, distributed computing partnership for data-intensive research. PATh Partnership to Advance Throughput Computing, bringing together CHTC and OSG t... National Research Platform A community-owned research and education platform for sharing compute, data, ... IRIS-HEP An institute advancing software R&D for the HL-LHC era of high energy physics... GP-ENGINE Great Plains Extended Network of GPUs for Interactive Experimenters. GP-STOR Great Plains Scalable Tiered Object Repository providing distributed, resilie... SciTokens Federated, capability-based authorization for distributed scientific computing.
// expertise
Technologies
- Kubernetes
- Docker
- Go
- Python
- React
- Node.js
- Next.js
- Git
- GitHub
- GitHub Copilot
- Tailwind CSS
- CSS3
- HTML5
- Vercel
- Flux
- Astro
- Nextcloud
- Falco
- Helm
- OpenSearch
- Elasticsearch
- Linux
- NVIDIA
- JWT
- Kubernetes
- Docker
- Go
- Python
- React
- Node.js
Show all technologies (24)
- Next.js
- Git
- GitHub
- GitHub Copilot
- Tailwind CSS
- CSS3
- HTML5
- Vercel
- Flux
- Astro
- Nextcloud
- Falco
- Helm
- OpenSearch
- Elasticsearch
- Linux
- NVIDIA
- JWT
// writing
Blog
Profiling the XRootD Monitoring Collector
The XRootD Monitoring Collector (collector) receives file transfer accounting messages from XRootD servers.This transfer info...
Jan 31, 2024
Dashboards for Learning Data Visualizations
Creating dashboards and data visualizations are a favorite past time of mine. Also, I jump at any chance to learn a new tech...
Sep 14, 2022
StashCache
StashCache is a framework to distribute data across the Open Science Grid. It is designed to help opportunistic users to tra...
Jun 14, 2017
Running R at HCC
The full presentation.There are many methods to run R applications at HCC. I can break these uses down to: Creating a tradi...
Sep 21, 2016
Hiding all the details: Grid jobs in Docker
Table of Contents For a long time, HTCondor has strived to have the job runtime environment be run and defined by the s...
May 18, 2016