Profiling the XRootD Monitoring Collector
The XRootD Monitoring Collector (collector) receives file transfer accounting messages from XRootD servers. This transfer information is parsed by the collec...
Posts by Year
2024
2022
Dashboards for Learning Data Visualizations
Creating dashboards and data visualizations are a favorite past time of mine. Also, I jump at any chance to learn a new technology. That is why I have spen...
Improving the Open Science Data Federation’s Cache Selection
Optimizing data transfers requires tuning many parameters. High latency between the client and a server can decrease data transfer throughput. The Open Scie...
2020
XRootD Client Manager
The validation project for XRootD Monitoring is moving to phase 2, scale testing. Phase 1 focused on correctness of single server monitoring. The report is...
GRACC Transition Visualization
The OSG is in the progress of transitioning from an older ElasticSearch (ES) cluster to a new version. Part of this process is reindexing (copying) data fro...
2019
LetsEncrypt for Multiple Hosts
Using LetsEncrypt for certificate creation and management has made secure communications much easier. Instead of contacting the IT department of your univer...
2018
StashCache By The Numbers
The StashCache federation is comprised of 3 components: Origins, Caches, and Clients. There are additional components that increase the usability of StashCa...
HTCondor Pull Mode
For a recent project to utilize HPC clusters for HTC workflows, I had to add the ability to transfer the input and output sandboxes to and from HTCondor. HT...
2017
Cleaning Up GRACC
The GRid ACcounting Collector (GRACC) is the OSG’s new version of accounting software, replacing Gratia. It has been running in production since March 2017....
Installing SciTokens on a Mac
In case I ever have to install SciTokens again, the steps I took to make it work on my mac. The most difficult part of this is installing openssl headers fo...
StashCache
StashCache is a framework to distribute data across the Open Science Grid. It is designed to help opportunistic users to transfer data without the need for ...
Deploying Docs on Github with Travis-CI
It is very common to deploy docs from a Github repo to a Github Pages site. In the past few days, I have setup several repos that will push to Github Pages...
Singularity on the OSG
Singularity is a container platform designed for use on computational resources. Several sites have deployed Singularity for their users and the OSG. In thi...
2016
Running Mixed EL6 & EL7 Clusters
We are entering a new era of transition from Enterprise Linux (EL) 6 to EL7. During this transition, we have to support submitting jobs to clusters that are...
Running R at HCC
The full presentation. There are many methods to run R applications at HCC. I can break these uses down to: Creating a traditional Slurm submit file th...
Hiding all the details: Grid jobs in Docker
Table of Contents For a long time, HTCondor has strived to have the job runtime environment be run and defined by the submit host. But, that is...
Building CentOS packages on Travis-CI
The Travis-CI Continuous Integration service is great for building and testing software for each commit. But, it is limited to only supporting builds and ...
HTCondor-CE-Bosco Upcoming Release
The HTCondor-CE-Bosco (CE-Bosco) is one of the largest changes for the upcoming OSG 3.3.12 release, to be released on 2016/05/10. The HTCondor-CE-Bosco is a...
A New Blogging Platform
I have decided to move my Blog from Blogspot to Github Pages, and Jekyll publisher. I made this move for many reasons, but my latest blog post showed how...
Querying an Elasticsearch Cluster for Gratia Records
For the last few days I have been working on email reports for GRACC, OSG’s new prototype accounting system. The source of the email reports are located on ...
Fedora copr: Slurm per job tmp directories!
A Slurm site on the OSG was having problems with /tmp filling up occasionally. Our own clusters have a Slurm Spank plugin that creates a per job tmp directo...
2015
The StashCache Tester
StashCache is is a framework to distribute user data from an origin site to an job. It uses a layer of caching, as well as the high performance XRootD servi...
GPUs and adding new resources types to the HTCondor-CE
In the past, it has been difficult to add new resource types to the OSG CE, whether it was a Globus GRAM CE or a HTCondor-CE. But, it has just gotten a litt...
The more things change, the more they stay the same
A lot has happened since I last posted in January.<ol><li>I have successfully defended and submitted my dissertation: Enabling Distributed Scient...
HTCondor CacheD: Caching for HTC - Part 2
In the previous post, I discussed why we decided to make the HTCondor CacheD. This time, we will discuss the operation and design of the CacheD, as well as ...
Condor CacheD: Caching for HTC - Part 1
A typical job flow in the Open Science Grid is as follows:<ol><li>Stage input files to worker node.</li><li>Start processing…</li&...
2014
GPUs on the OSG
For a while, we have heard of the need for GPU’s on the Open Science Grid. Luckily, HCC hosts numerous GPU resources that we are willing to share. And wit...
Part 1: Features of the Chrome App - Embedding
In my previous post, I introduced my OSG Chrome App. In this post, I want to introduce one feature of the chrome app, Embedding profiles.<div></div...
OSG Usage Chrome App
Hi, I’m Derek Weitzel. You may remember me from such side projects as:<ul><li>OSG Android App</li><li>Redesign of Gratiaweb Interfac...
A HCC Dashboard with OSG Accounting
After the 2013 SuperComputing Conference, we found ourselves with a extra monitor at HCC. Therefore, I set about creating a dashboard which can show the cur...
Moving from a Globus to an HTCondor Compute Element
A few weeks ago, we moved our opportunistic clusters, Crane and Tusker, from Globus GRAM gatekeepers to the new HTCondor-CE. We moved to the HTCondor-CE i...
2013
OSG at Super Computing!
Nebraska hosted a booth at the Super Computing conference this year.<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" sty...
Bosco @ CHEP
This week, I attended the Computing in High Energy Physics (CHEP) 2013 conference in Amsterdam to present a poster on Bosco.<table align="center" cellpadd...
The NSA vs. my grid certificate
Note: Opinions or statements made in this post are solely my own, and are not the opinions of any organization that I work with or for. Also, I probably got...
You asked, we delivered!
A few users of BoscoR, and specifically GridR, have requested several major features:<ol><li>Allow custom packages to be installed on the remote...
How you know you are doing something right?
I just received this from Dropbox:<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-righ...
CIC 13 Meeting
This week, the Campus Infrastructures Community debuted the OSG Connect service. Overall it was a very successful workshop. I met 2 users that where very e...
Bosco & OSG @ XSEDE13
Last week, several members of the Bosco team attended the XSEDE 13 Conference. We enjoyed the great San Diego weather, and even caught a little ComicCon, wh...
Creating a native Mac Installer for Bosco
It has been our goal for some time to create a native installer for our supported platforms. This week, we created a prototype for the first of those platfo...
The next step for Bosco: BoscoR
With the 1.1 release of Bosco in January, the Bosco team finally had a usable product for researchers. It didn’t have the best usability, but it was a solid...
HCC OSG Workshop
Over the last 2 days, HCC has been running a workshop on:<ol><li>Building an HTC Cluster</li><li>Running on a HTC Cluster</li>&...
Bosco at GPN Conference
This week, I presented the Bosco poster at the Great Plains Network annual meeting. It received a very good reception. Many people where interested in wha...
Submitting R jobs with Bosco
The Bosco team has been working on integrating with the R statistics processing language. We have chosen to modify the GridR package in order to integrate w...
Reprocessing CMS events with Bosco
Prior to the LHC long shutdown, the CMS experiment increased the trigger rate of the detector, therefore increasing the data coming off the detector. The Ti...
Running Quantum Espresso on the OSG
While running Quantum Espresso on the Open Science Grid, we found a number of issues:<ul><li>OpenMPI needs to have an rsh binary. Even if you ar...
Using Bosco to submit to Amazon EC2
A homework assignment for my storage class required running ~30 hours of benchmarks of the btrfs and ext4 filesystems. I thought this would be an excellent ...
Rendering with Bosco
Example image rendered using HCC's distributed renderer At the Nebraska Holland Computing Center, we take pride in eating our own dog food. So I want to hi...
Bosco 1.1.1 Release
Today I am pleased to announce that Bosco version 1.1.1 has been released. This is a patch fix for the 1.1 to address 2 issues that where affecting users. T...
Improving Gratia’s Web Interface
Over the winter break, I worked on improving the interface that most users use for OSG accounting. When I returned from break, I worked with Ashu to integra...
Bosco 1.1 Release
In the last few months, I have outlined the new features of Bosco 1.1.<ul><li>Mutli-Cluster Support</li><li>Multi-OS Support</li&...
2012
BOSCO v1.1 Features: Multi-Cluster Support
This is the forth in the in the series of features in the 1.1 release of BOSCO. The previous posts have focused on SSH File Transfer, Single Port Usage, and...
BOSCO v1.1 Features: Multi-OS Support
This is part 3 of my ongoing series describing new features in BOSCO v1.1. Part 1 covered file transfer over SSH, Part 2 covered single port usage. This po...
BOSCO v1.1 Features: Single Port Usage
Welcome to part 2 of my ongoing series of v1.1 features for BOSCO. Part 1 was on SSH File Transfer.This time, I’ll talk about a new feature that we didn’t p...
BOSCO v1.1 Features: SSH File Transfer
I am hoping to write about a few of the new features of BOSCO 1.1 before it comes out in December or January. This is part 1 of that series:<h3>BOSCO ...
The Island Grid: Puerto Rico
In the last post, I discussed my trip to Puerto Rico. Now that my trip is over, and I am back in Lincoln, I would like to share some of the successes of the...
Building a Campus Grid on a Island
Hello everyone from San Juan, Puerto Rico. <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; ma...
Flocking to the OSG behind a restrictive Firewall
On many campuses, restrictive firewalls are the name of the game, rather then the exception. Here at Nebraska, we have the ability to define our own network...
CMS with the Campus Factory
The Campus Factory is usually used by small research groups to expand their available resources to those on the campus. Of course, that’s not always easy f...
Day 4: Open Science Grid Summer School 2012
Yesterday, I taught the class storage on the OSG. Since I was teaching, I was unable to write or take any pictures.Today focuses on actual science on the OS...
Video of Igor’s Exercise
Igor had a very interesting exercise during the OSG Summer School. It’s a little difficult to explain, but:Each student at the tables is a worker node. The ...
Day 2 of OSG Summer School
We had a great day yesterday at the OSG Summer School. Not only was the weather great, but the exercises went very well (a testament to Alain). Monday...
Summer School Pictures
I’m putting pictures from the summer school on my Google Plus.
Day 1 of OSG Summer School
I was asked to teach at the OSG Summer School. I think teaching the next generation of OSG users is a great opportunity. This is how I learned about the OS...
Installing and Configuring glusterfs on EL6
I’m always interested in the newest technologies. With the purchase of Gluster by RedHat, I figured it was time to give it a try. We are always looking for...
Installing your own gratia webpage
In the OSG, we use a technology called Gratia for our accounting. Every single job and data transfer on the OSG is accounted for in the database. Of course...
In NY for CHEP
Hanging out in Times Square I’m in New York this week for CHEP 2012. I’ll be writing about my experiences here. Stay Tuned.
Developments at Nebraska
I thought I would do a quick post about recent developments at Nebraska.Tusker & OSG<table align="center" cellpadding="0" cellspacing="0" class="tr-ca...
BOSCO + Campus Factory
Checklist while implementing CF + Bosco integration For the last several months, the campus infrastructure team has worked on software that will help users c...
OSG AHM 2012
This years all hands meeting was a great success! There where a few sessions that I really enjoyed.AHM Pictures Campus CaucusThere where many user engageme...
Burning the LiveUSBs
In my last post, I talked about the OSG LiveUSBs. Now that the conference is next week, I have started burning the USB with the image. <table align="cent...
Building an OSG-Client LiveUSB
Nebraska/OSG USB keys to be distributed at OSG-AHM 2012 Since we have started using RPM’s for osg software, I’ve been interested if it where possible to make...
Ceph on Fedora 16
I’ve written before how to run ceph on Fedora 15, but now I’m working on Fedora 16.Last time I complained about how much ceph tries to do for you. For bette...
Fedora 16 on OpenStack
After following Brian’s guide on installing Fedora 15 on OpenStack, I thought I would try my hand at Fedora 16. There where a few differences.Filesystem Dif...
Testing an Globus Free OSG-Software (From EPEL(-testing))
As you may or may not know, there is a massive globus update pending in EPEL that will update globus to the version the OSG distributes. What this means is ...
Initial EL6 Packages for OSG
Last night I completed initial packages for EL6 support. Just like for EL5, the first OSG component I created is the osg-wn-client.The osg-wn-client has a c...
2011
September Progress Report
Since September was a while ago, I’ll keep this short. Most of Sept. was figuring out the class/work/research schedule. It had been a 2 years since I’ve ta...
SC11 Visualization Prep
Last year I put together a visualization for SuperComputing 10’ to show jobs moving over the OSG.<div class="separator" style="clear: both; text-align: ce...
CEPH on Fedora 15
Yesterday, I read a blog post using CEPH for a backend store for virtual machine images. I’ve heard a lot about ceph in the last year, especially after it w...
OSG… There’s an app for that
So I thought this weekend would be a good time to learn something new. I put together an OSG Android App. Not all the features are working yet, but I tend ...
Upgrading the osg-client to Testing
Another lively blog post about upgrading a set of packages to testing. Last time it was the osg-wn-client, this time it’s the OSG-Client. I will use a jira...
Upgrading OSG Software from Development to Testing
In the OSG-software team, there has been a lot of discussion when the right time to upgrade packages to testing from the developer playground. I’m not going...
OSG Site Admin meeting: Day 2
Another beautiful (read: HOT) day in Lubbock.<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" styl...
OSG Site Admin meeting: Day 1
Hello from the hot, hot, hot Lubbock.<div class="separator" style="clear: both; text-align: center;"></div>Had a good morning of broad plenary ta...
Testing Release of OSG-Client RPM
It’s finally here! The first official testing release of the osg-client in RPM form. We’re asking anyone and everyone to go out there and start testing. T...
Job Visualization
A year or so ago, Ian Stokes-Rees showed me a job visualization that he had put together to see how his workflows where going. Gives a good overview per sit...
HCC Walltime Effeciency
Rob’s post on walltime efficiency was interesting. I did the same with HCC and found that we had much better efficiency than I expected. Especially with G...
Need more cores, Scotty!
A researcher/grid user from UNL has come back! Monitoring 100K+ Jobs. Even though we have a lot of jobs idle, we are only able to get maybe 4.5k runni...
Gratia DB Rates
In my recent discussions with NCSA on gratia, I found something that is probably obvious to other people, but I would still like to point out.<table align...
Using VOMS Groups and Roles
I’ve been working on gratia graphs with Ashu Guru for the new GlideinWMS probe. I’ve stumbled upon very odd command line usage of voms-proxy-init. <p>I...
Overflowing the Tier 2
The last few days, the Nebraska CMS Tier 2 has been full. And I mean really full.<div style="text-align: center;"></div>The expected wait time f...
Day 7 at CERN - Saturday
For breakfast, fruit and a croissant.Brian and I walked to the bus stop at CERN, tried deciphering the bus schedule. We found the correct bus (after walking...
Day 6 at CERN - Friday
Breakfast was orange juice, fruit, and a croissant.Morning SessionThis morning was the site related meetings. We had a talk from Oliver from FNAL regarding ...
Day 5 at CERN - Thursday
Breakfast was Choco Puffs, Fruit, and Orange/Passion fruit juice.Today was largely Management meetings, so I was free to play tourist at CERN. I went to the...
Day 4 at CERN - Wednesday
Breakfast was Milk, Orange juice, Fruit, Vanilla (Natural) yogurt, and croissant.Morning SessionThis morning is Monitoring Task Force. This stuff is very in...
Day 3 at CERN - Tuesday
This morning was mostly spent fixing my user registration and getting my CERN account. Breakfast was a slice of keish, fruit, and an orange juice.Morning Se...
Day 2 At CERN
Woke up this morning at 7:30. Grabbed some breakfast across the street from the Hostel. I had orange juice with passion fruit (whatever that is), mixed fru...
Arriving at CERN
Had a decent flight into CERN. Aaron D. picked Brian and I up at the airport. The Hostel opens at 9:00, so we ate some breakfast in the cafeteria before we...
Making a new group in GlideinWMS VO Frontend
The most difficult thing with the RPM (or the normal install) is making a new group in the GlideinWMS Frontend.The files you need to add are:<pre class="b...
Non-root RPM Install
First, copy the database files from rpm. This is so you don’t have to reinstall things already installed on the system.cp /var/lib/rpm/* newprefix/To instal...
2010
Google Earth Fedora 13 Fixed!
I was able to fix the install of google earth on fedora 13 by installing the i686 nvidia libraries.After adding rpm forge and installing the nvidia drivers (...
Security Visualization
</param></param></param></embed>
Condor Week
Just got back from condor week. I gave a presentation on persistent jobs in Condor. Check out the other presentations:
2009
Hadoop Viz
Something I’ve been working on.</param></param></param></embed>
Youtube Channel
On my youtube channel, you can view the visualizations I’m working on.YouTube Channel</param></param></param></embed>
Adding Routes at startup
To add routes to be started, the file located at /etc/sysconfig/network-scripts/ifup-routes will parse each file named route-eth*. The file should end up lo...
Senior Design Done
It’s the end of the semester, and that means the end of Senior Design. If you are interested in a Linux 3D immersive game using Wii motes, here’s the link.
XWarpPointer
Just came across a way to move the mouse pointer to an absolute position in X. Follow the code below:<pre class="code">int main(){ Display* xdi...
Parsing HTTP Responses
Just so I don’t have to remember again, in order to seperate a header from the body of a http response, in C:<pre name="code" class="Bash">char* endOfH...
No devices: No Screens Detected.
Had a problem today with the no screens detected on the new Viz wall computers. Solved it by first installing the Nvidia binaries straight from their websit...
Fuse DFS Error
So, today I was having problems getting the fuse-dfs to work with my cluster. I was an error compiling fuse for my kernel (2.6.18-92.1.17.el5). It seems th...
Fish Monitoring
After a few years of working on the Fish, I have finally published a video of them running, so that they could be demonstrated easier.</embed>
Viz Wall Computers
Today I put together a computer spec for the viz wall we are building here at RCF.Here’s the link to the page:http://docs.google.com/Doc?id=dgjdk82m_23vp8rwfc2