Feb 13, 2026
Debugging OpenSearch Transform jobs can feel like searching for a needle in a haystack, especially when the error messages are generic. This post chronicles a recent debugging journey, highlighting common pi...
Jan 31, 2024
The XRootD Monitoring Collector (collector) receives file transfer accounting messages from XRootD servers.This transfer information is parsed by the collector and sent to the GRACC accounting database for v...
Sep 14, 2022
Creating dashboards and data visualizations are a favorite past time of mine. Also, I jump at any chance to learn a new technology. That is why I have spent the last couple of months building dashboards an...
Jan 22, 2022
Optimizing data transfers requires tuning many parameters. High latency between the client and a server can decrease data transfer throughput. The Open Science Data Federation (OSDF) attempts to optimize th...
Oct 11, 2020
The validation project for XRootD Monitoring is moving to phase 2, scaletesting. Phase 1 focused on correctness of single server monitoring. Thereport is available.
Mar 08, 2020
The OSG is in the progress of transitioning from an older ElasticSearch (ES) cluster to a new version. Part of this process is reindexing (copying) data from the old to the new. Unfortunately, it’s not eas...
Oct 11, 2019
Using LetsEncrypt for certificate creation and management has made secure communications much easier. Instead of contacting the IT department of your university to request a certificate, you can skip the mi...
Sep 26, 2018
The StashCache federation is comprised of 3 components: Origins, Caches, and Clients. There are additional components that increase the usability of StashCache which I will also mention in this post.
Aug 31, 2018
For a recent project to utilize HPC clusters for HTC workflows, I had to add the ability to transfer the input and output sandboxes to and from HTCondor. HTCondor already has the ability to spool input file...
Nov 06, 2017
The GRid ACcounting Collector (GRACC) is the OSG’s new version of accounting software, replacing Gratia. It has been running in production since March 2017. Last week, on Friday November 3rd, we held a GRA...
Sep 07, 2017
In case I ever have to install SciTokens again, the steps I took to make it work on my mac. The most difficult part of this is installing openssl headers for the jwt python library. I followed the advice o...
Jun 14, 2017
StashCache is a framework to distribute data across the Open Science Grid. It is designed to help opportunistic users to transfer data without the need for dedicated storage or frameworks of their own, like...
Feb 08, 2017
It is very common to deploy docs from a Github repo to a Github Pages site. In the past few days, I have setupseveral repos that will push to Github Pages using the Travis-CI continuous integration, and I w...
Jan 12, 2017
Singularity is a container platform designed for use on computational resources.Several sites have deployed Singularity for their users and the OSG. In this post, I will provided a tutorial on how to usesin...
Nov 11, 2016
We are entering a new era of transition from Enterprise Linux (EL) 6 to EL7. During this transition, we have to support submitting jobs to clusters that are running one or both of these OS’s. In this post,...
Sep 21, 2016
The full presentation.There are many methods to run R applications at HCC. I can break these uses down to: Creating a traditional Slurm submit file that runs an R script. The vast majority of R users do t...
May 18, 2016
Table of Contents For a long time, HTCondor has strived to have the job runtime environment be run and defined by the submit host. But, that is surprisingly difficult to do. There are many reasons wh...
May 03, 2016
The Travis-CI Continuous Integration service is great for building and testing software for each commit. But, it is limited to only supporting builds and tests on the Ubuntu OS. The OSG, on the other hand,...
Apr 26, 2016
The HTCondor-CE-Bosco (CE-Bosco) is one of the largest changes for the upcoming OSG 3.3.12 release, to be released on 2016/05/10. The HTCondor-CE-Bosco is a special configuration of the HTCondor-CE. The HT...
Apr 21, 2016
I have decided to move my Blog from Blogspot to Github Pages, and Jekyll publisher. I made this move for many reasons, but my latest blog post showed how difficult it can be to create technical posts in Blo...
Apr 20, 2016
For the last few days I have been working on email reports for GRACC, OSG’s new prototype accounting system. The source of the email reports are located on Github.I have learned a significant amount about q...
Mar 22, 2016
A Slurm site on the OSG was having problems with /tmp filling up occasionally. Our own clusters have a Slurm Spank plugin that creates a per job tmp directory. This per job directory is useful for grid job...
Oct 14, 2015
StashCache is is a framework to distribute user data from an origin site to an job. It uses a layer of caching, as well as the high performance XRootD service in order to distribute the data. It can source...
Aug 07, 2015
In the past, it has been difficult to add new resource types to the OSG CE, whether it was a Globus GRAM CE or a HTCondor-CE. But, it has just gotten a little bit easier. Today I added HCC’s GPU resources ...
Jul 28, 2015
A lot has happened since I last posted in January.<ol><li>I have successfully defended and submitted my dissertation: Enabling Distributed Scientific Computing on the Campus. I will formally gra...
Jan 25, 2015
In the previous post, I discussed why we decided to make the HTCondor CacheD. This time, we will discuss the operation and design of the CacheD, as well as show an example utilizing a BLAST database.It is i...
Jan 22, 2015
A typical job flow in the Open Science Grid is as follows:<ol><li>Stage input files to worker node.</li><li>Start processing…</li><li>Stage output files back to submit hos...
Jun 25, 2014
For a while, we have heard of the need for GPU’s on the Open Science Grid. Luckily, HCC hosts numerous GPU resources that we are willing to share. And with the release today of HTCondor 8.2, we wanted to i...
Apr 09, 2014
In my previous post, I introduced my OSG Chrome App. In this post, I want to introduce one feature of the chrome app, Embedding profiles.<div></div><div>Part of my morning routine is check...
Apr 04, 2014
Hi, I’m Derek Weitzel. You may remember me from such side projects as:<ul><li>OSG Android App</li><li>Redesign of Gratiaweb Interface</li><li>Native Mac Installer for Bos...
Mar 12, 2014
After the 2013 SuperComputing Conference, we found ourselves with a extra monitor at HCC. Therefore, I set about creating a dashboard which can show the current status of HCC.<div class="separator" style...
Feb 27, 2014
A few weeks ago, we moved our opportunistic clusters, Crane and Tusker, from Globus GRAM gatekeepers to the new HTCondor-CE. We moved to the HTCondor-CE in order to solve performance issues we experienced w...
Nov 25, 2013
Nebraska hosted a booth at the Super Computing conference this year.<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-al...
Oct 18, 2013
This week, I attended the Computing in High Energy Physics (CHEP) 2013 conference in Amsterdam to present a poster on Bosco.<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-containe...
Sep 10, 2013
Note: Opinions or statements made in this post are solely my own, and are not the opinions of any organization that I work with or for. Also, I probably got some details wrong… so there’s that.Though I’m no...
Aug 30, 2013
A few users of BoscoR, and specifically GridR, have requested several major features:<ol><li>Allow custom packages to be installed on the remote worker node (#3).</li><li>Specify cust...
Aug 30, 2013
I just received this from Dropbox:<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><...
Aug 29, 2013
This week, the Campus Infrastructures Community debuted the OSG Connect service. Overall it was a very successful workshop. I met 2 users that where very excited about BoscoR, and we had great discussions ...
Jul 29, 2013
Last week, several members of the Bosco team attended the XSEDE 13 Conference. We enjoyed the great San Diego weather, and even caught a little ComicCon, which was being held next door when I arrived. Here...
Jul 11, 2013
It has been our goal for some time to create a native installer for our supported platforms. This week, we created a prototype for the first of those platforms, the Mac installer.<h3>The PKG file</...
Jul 05, 2013
With the 1.1 release of Bosco in January, the Bosco team finally had a usable product for researchers. It didn’t have the best usability, but it was a solid tool to build on. When planning the next release...
Jun 05, 2013
Over the last 2 days, HCC has been running a workshop on:<ol><li>Building an HTC Cluster</li><li>Running on a HTC Cluster</li></ol><h3>Building an HTC Cluster</h3...
May 31, 2013
This week, I presented the Bosco poster at the Great Plains Network annual meeting. It received a very good reception. Many people where interested in what Bosco could do for them. Most of the audience wa...
May 20, 2013
The Bosco team has been working on integrating with the R statistics processing language. We have chosen to modify the GridR package in order to integrate with R.<h3>How will the R user see Bosco?<...
Apr 02, 2013
Prior to the LHC long shutdown, the CMS experiment increased the trigger rate of the detector, therefore increasing the data coming off the detector. The Tier-0 was unable to process all of the events comin...
Mar 05, 2013
While running Quantum Espresso on the Open Science Grid, we found a number of issues:<ul><li>OpenMPI needs to have an rsh binary. Even if you are using shared memory for openmpi, and openmpi doe...
Feb 06, 2013
A homework assignment for my storage class required running ~30 hours of benchmarks of the btrfs and ext4 filesystems. I thought this would be an excellent time to test Bosco’s ability to submit to Amazon E...
Jan 16, 2013
Example image rendered using HCC's distributed rendererAt the Nebraska Holland Computing Center, we take pride in eating our own dog food. So I want to highlight one of our uses of Bosco to enable transpare...
Jan 14, 2013
Today I am pleased to announce that Bosco version 1.1.1 has been released. This is a patch fix for the 1.1 to address 2 issues that where affecting users. The release is available on the Bosco Download Page...
Jan 14, 2013
Over the winter break, I worked on improving the interface that most users use for OSG accounting. When I returned from break, I worked with Ashu to integrate the changes with some recent changes he had mad...
Jan 08, 2013
In the last few months, I have outlined the new features of Bosco 1.1.<ul><li>Mutli-Cluster Support</li><li>Multi-OS Support</li><li>Single Port Usage</li><li>...
Nov 29, 2012
This is the forth in the in the series of features in the 1.1 release of BOSCO. The previous posts have focused on SSH File Transfer, Single Port Usage, and Multi-OS Support. All of these features are new ...
Nov 06, 2012
This is part 3 of my ongoing series describing new features in BOSCO v1.1. Part 1 covered file transfer over SSH, Part 2 covered single port usage. This post will cover the Multi-OS Support in BOSCO.<h3...
Nov 01, 2012
Welcome to part 2 of my ongoing series of v1.1 features for BOSCO. Part 1 was on SSH File Transfer.This time, I’ll talk about a new feature that we didn’t planned on implementing at first, using only a sing...
Oct 29, 2012
I am hoping to write about a few of the new features of BOSCO 1.1 before it comes out in December or January. This is part 1 of that series:<h3>BOSCO v1.1 Feature: SSH File Transfer</h3><div&...
Oct 18, 2012
In the last post, I discussed my trip to Puerto Rico. Now that my trip is over, and I am back in Lincoln, I would like to share some of the successes of the trip. I had a great time exploring old San Juan....
Oct 15, 2012
Hello everyone from San Juan, Puerto Rico. <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbod...
Sep 28, 2012
On many campuses, restrictive firewalls are the name of the game, rather then the exception. Here at Nebraska, we have the ability to define our own network policies. But some universities / labs restrict ...
Jun 29, 2012
The Campus Factory is usually used by small research groups to expand their available resources to those on the campus. Of course, that’s not always easy for larger VO’s, who tend to have more complicated s...
Jun 28, 2012
Yesterday, I taught the class storage on the OSG. Since I was teaching, I was unable to write or take any pictures.Today focuses on actual science on the OSG. First, how actual science runs on the OSG with...
Jun 26, 2012
Igor had a very interesting exercise during the OSG Summer School. It’s a little difficult to explain, but:Each student at the tables is a worker node. The people walking around are the ‘network’, sending j...
Jun 26, 2012
We had a great day yesterday at the OSG Summer School. Not only was the weather great, but the exercises went very well (a testament to Alain).Monday Evening Work SectionThe evening work section was gr...
Jun 25, 2012
I’m putting pictures from the summer school on my Google Plus.
Jun 25, 2012
I was asked to teach at the OSG Summer School. I think teaching the next generation of OSG users is a great opportunity. This is how I learned about the OSG, at the International Summer School for Grid Com...
Jun 02, 2012
I’m always interested in the newest technologies. With the purchase of Gluster by RedHat, I figured it was time to give it a try. We are always looking for new technology that can lower the operations of o...
May 29, 2012
In the OSG, we use a technology called Gratia for our accounting. Every single job and data transfer on the OSG is accounted for in the database. Of course, this is a lot of data, ~500,000 jobs and 1.2M tr...
May 21, 2012
Hanging out in Times SquareI’m in New York this week for CHEP 2012. I’ll be writing about my experiences here. Stay Tuned.
Apr 20, 2012
I thought I would do a quick post about recent developments at Nebraska.Tusker & OSG<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right:...
Apr 16, 2012
Checklist while implementing CF + Bosco integrationFor the last several months, the campus infrastructure team has worked on software that will help users create a larger, more inclusive campus grid. The go...
Mar 27, 2012
This years all hands meeting was a great success! There where a few sessions that I really enjoyed.AHM Pictures Campus CaucusThere where many user engagement people there. I believe that we reached a conse...
Mar 16, 2012
In my last post, I talked about the OSG LiveUSBs. Now that the conference is next week, I have started burning the USB with the image. <table align="center" cellpadding="0" cellspacing="0" class="tr-capt...
Feb 24, 2012
Nebraska/OSG USB keys to be distributed at OSG-AHM 2012Since we have started using RPM’s for osg software, I’ve been interested if it where possible to make a LiveUSB of the client. Due to the great documen...
Feb 17, 2012
I’ve written before how to run ceph on Fedora 15, but now I’m working on Fedora 16.Last time I complained about how much ceph tries to do for you. For better or worse, now it attempts to do more for you!For...
Feb 07, 2012
After following Brian’s guide on installing Fedora 15 on OpenStack, I thought I would try my hand at Fedora 16. There where a few differences.Filesystem DifferencesBrian’s guide installed Fedora using LVM. ...
Jan 24, 2012
As you may or may not know, there is a massive globus update pending in EPEL that will update globus to the version the OSG distributes. What this means is much less work for the osg-software team since we ...
Jan 20, 2012
Last night I completed initial packages for EL6 support. Just like for EL5, the first OSG component I created is the osg-wn-client.The osg-wn-client has a complicated dependency tree. Easily some of the mo...
Oct 25, 2011
Since September was a while ago, I’ll keep this short. Most of Sept. was figuring out the class/work/research schedule. It had been a 2 years since I’ve taken anywhere near a full load of classes, and it’s...
Oct 21, 2011
Last year I put together a visualization for SuperComputing 10’ to show jobs moving over the OSG.<div class="separator" style="clear: both; text-align: center;"></div>This visualization has since...
Oct 14, 2011
Yesterday, I read a blog post using CEPH for a backend store for virtual machine images. I’ve heard a lot about ceph in the last year, especially after it was integrated into the mainline kernel in 2.6.34. ...
Sep 13, 2011
So I thought this weekend would be a good time to learn something new. I put together an OSG Android App. Not all the features are working yet, but I tend to release early and often on all my projects.I wa...
Aug 15, 2011
Another lively blog post about upgrading a set of packages to testing. Last time it was the osg-wn-client, this time it’s the OSG-Client. I will use a jira task to track my time and progress on it.Again, I...
Aug 14, 2011
In the OSG-software team, there has been a lot of discussion when the right time to upgrade packages to testing from the developer playground. I’m not going to go into the discussion much, other than it sho...
Aug 10, 2011
Another beautiful (read: HOT) day in Lubbock.<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div&g...
Aug 09, 2011
Hello from the hot, hot, hot Lubbock.<div class="separator" style="clear: both; text-align: center;"></div>Had a good morning of broad plenary talks. The indico schedule gives links to the talk...
Aug 04, 2011
It’s finally here! The first official testing release of the osg-client in RPM form. We’re asking anyone and everyone to go out there and start testing. The official instructions are on the OSG Twiki. We...
Jun 17, 2011
A year or so ago, Ian Stokes-Rees showed me a job visualization that he had put together to see how his workflows where going. Gives a good overview per site. Recently, we where investigating usage from a ...
Jun 11, 2011
Rob’s post on walltime efficiency was interesting. I did the same with HCC and found that we had much better efficiency than I expected. Especially with GlideinWMS’s default behavior of sticking around for...
Jun 10, 2011
A researcher/grid user from UNL has come back! Monitoring100K+ Jobs.Even though we have a lot of jobs idle, we are only able to get maybe 4.5k running jobs. Doing some investigating, it looks lik...
Jun 09, 2011
In my recent discussions with NCSA on gratia, I found something that is probably obvious to other people, but I would still like to point out.<table align="center" cellpadding="0" cellspacing="0" class="t...
Jun 07, 2011
I’ve been working on gratia graphs with Ashu Guru for the new GlideinWMS probe. I’ve stumbled upon very odd command line usage of voms-proxy-init.<p>In order to get a role to work with voms-proxy-init,...
Apr 28, 2011
The last few days, the Nebraska CMS Tier 2 has been full. And I mean really full.<div style="text-align: center;"></div>The expected wait time for some of these users (55 active users) is over a...
Apr 09, 2011
For breakfast, fruit and a croissant.Brian and I walked to the bus stop at CERN, tried deciphering the bus schedule. We found the correct bus (after walking to all 3 bus stops at CERN), and rode it to the T...
Apr 08, 2011
Breakfast was orange juice, fruit, and a croissant.Morning SessionThis morning was the site related meetings. We had a talk from Oliver from FNAL regarding data operations. It concluded that we should sepa...
Apr 07, 2011
Breakfast was Choco Puffs, Fruit, and Orange/Passion fruit juice.Today was largely Management meetings, so I was free to play tourist at CERN. I went to the Reception area:There I bought stuff for the famil...
Apr 06, 2011
Breakfast was Milk, Orange juice, Fruit, Vanilla (Natural) yogurt, and croissant.Morning SessionThis morning is Monitoring Task Force. This stuff is very interesting. I’m noticing a lot of CouchDB in the t...
Apr 05, 2011
This morning was mostly spent fixing my user registration and getting my CERN account. Breakfast was a slice of keish, fruit, and an orange juice.Morning SessionI attended a morning session on the new Data ...
Apr 04, 2011
Woke up this morning at 7:30. Grabbed some breakfast across the street from the Hostel. I had orange juice with passion fruit (whatever that is), mixed fruit, and a chocolate french roll.After this, Brian ...
Apr 03, 2011
Had a decent flight into CERN. Aaron D. picked Brian and I up at the airport. The Hostel opens at 9:00, so we ate some breakfast in the cafeteria before we checked into our hostel.After checking in, I took...
Feb 17, 2011
The most difficult thing with the RPM (or the normal install) is making a new group in the GlideinWMS Frontend.The files you need to add are:<pre class="brush: bash">cp -r /usr/share/gwms-frontend/www/...
Jan 29, 2011
First, copy the database files from rpm. This is so you don’t have to reinstall things already installed on the system.cp /var/lib/rpm/* newprefix/To install the rpms:rpm –prefix $HOME/newprefix -ivh rpmbui...
Oct 01, 2010
I was able to fix the install of google earth on fedora 13 by installing the i686 nvidia libraries.After adding rpm forge and installing the nvidia drivers (kmod-nvidia). It only installed the x86_64 nvidia...
Apr 22, 2010
Just got back from condor week. I gave a presentation on persistent jobs in Condor. Check out the other presentations:
Dec 23, 2009
Something I’ve been working on.</param></param></param></embed>
Jul 02, 2009
On my youtube channel, you can view the visualizations I’m working on.YouTube Channel</param></param></param></embed>
May 14, 2009
To add routes to be started, the file located at /etc/sysconfig/network-scripts/ifup-routes will parse each file named route-eth*. The file should end up looking like:<pre>ADDRESS0=192.168.2.0NETMASK0...
Apr 29, 2009
It’s the end of the semester, and that means the end of Senior Design. If you are interested in a Linux 3D immersive game using Wii motes, here’s the link.
Apr 29, 2009
Just came across a way to move the mouse pointer to an absolute position in X. Follow the code below:<pre class="code">int main(){ Display* xdisplay1 = XOpenDisplay(NULL); if(!xdisplay1)...
Feb 25, 2009
Just so I don’t have to remember again, in order to seperate a header from the body of a http response, in C:<pre name="code" class="Bash">char* endOfHeader = strstr(buf, “\n\r\n”);</pre>Notice t...
Feb 17, 2009
Had a problem today with the no screens detected on the new Viz wall computers. Solved it by first installing the Nvidia binaries straight from their website, and then allowing X configure itself:<pre cl...
Feb 12, 2009
So, today I was having problems getting the fuse-dfs to work with my cluster. I was an error compiling fuse for my kernel (2.6.18-92.1.17.el5). It seems that yum has stop providing this kernel, and it’s he...
Feb 10, 2009
After a few years of working on the Fish, I have finally published a video of them running, so that they could be demonstrated easier.</embed>
Feb 09, 2009
Today I put together a computer spec for the viz wall we are building here at RCF.Here’s the link to the page:http://docs.google.com/Doc?id=dgjdk82m_23vp8rwfc2