Posts by Year

2024

Profiling the XRootD Monitoring Collector

4 minute read

The XRootD Monitoring Collector (collector) receives file transfer accounting messages from XRootD servers. This transfer information is parsed by the collec...

2022

Dashboards for Learning Data Visualizations

3 minute read

Creating dashboards and data visualizations are a favorite past time of mine. Also, I jump at any chance to learn a new technology. That is why I have spen...

2020

XRootD Client Manager

3 minute read

The validation project for XRootD Monitoring is moving to phase 2, scale testing. Phase 1 focused on correctness of single server monitoring. The report is...

GRACC Transition Visualization

2 minute read

The OSG is in the progress of transitioning from an older ElasticSearch (ES) cluster to a new version. Part of this process is reindexing (copying) data fro...

2019

LetsEncrypt for Multiple Hosts

1 minute read

Using LetsEncrypt for certificate creation and management has made secure communications much easier. Instead of contacting the IT department of your univer...

2018

StashCache By The Numbers

3 minute read

The StashCache federation is comprised of 3 components: Origins, Caches, and Clients. There are additional components that increase the usability of StashCa...

HTCondor Pull Mode

1 minute read

For a recent project to utilize HPC clusters for HTC workflows, I had to add the ability to transfer the input and output sandboxes to and from HTCondor. HT...

2017

Cleaning Up GRACC

2 minute read

The GRid ACcounting Collector (GRACC) is the OSG’s new version of accounting software, replacing Gratia. It has been running in production since March 2017....

Installing SciTokens on a Mac

less than 1 minute read

In case I ever have to install SciTokens again, the steps I took to make it work on my mac. The most difficult part of this is installing openssl headers fo...

StashCache

4 minute read

StashCache is a framework to distribute data across the Open Science Grid. It is designed to help opportunistic users to transfer data without the need for ...

Deploying Docs on Github with Travis-CI

3 minute read

It is very common to deploy docs from a Github repo to a Github Pages site. In the past few days, I have setup several repos that will push to Github Pages...

Singularity on the OSG

3 minute read

Singularity is a container platform designed for use on computational resources. Several sites have deployed Singularity for their users and the OSG. In thi...

2016

Running Mixed EL6 & EL7 Clusters

2 minute read

We are entering a new era of transition from Enterprise Linux (EL) 6 to EL7. During this transition, we have to support submitting jobs to clusters that are...

Running R at HCC

3 minute read

The full presentation. There are many methods to run R applications at HCC. I can break these uses down to: Creating a traditional Slurm submit file th...

Building CentOS packages on Travis-CI

3 minute read

The Travis-CI Continuous Integration service is great for building and testing software for each commit. But, it is limited to only supporting builds and ...

HTCondor-CE-Bosco Upcoming Release

4 minute read

The HTCondor-CE-Bosco (CE-Bosco) is one of the largest changes for the upcoming OSG 3.3.12 release, to be released on 2016/05/10. The HTCondor-CE-Bosco is a...

A New Blogging Platform

less than 1 minute read

I have decided to move my Blog from Blogspot to Github Pages, and Jekyll publisher. I made this move for many reasons, but my latest blog post showed how...

Fedora copr: Slurm per job tmp directories!

less than 1 minute read

A Slurm site on the OSG was having problems with /tmp filling up occasionally.  Our own clusters have a Slurm Spank plugin that creates a per job tmp directo...

2015

The StashCache Tester

1 minute read

StashCache is is a framework to distribute user data from an origin site to an job.  It uses a layer of caching, as well as the high performance XRootD servi...

HTCondor CacheD: Caching for HTC - Part 2

4 minute read

In the previous post, I discussed why we decided to make the HTCondor CacheD.  This time, we will discuss the operation and design of the CacheD, as well as ...

Condor CacheD: Caching for HTC - Part 1

6 minute read

A typical job flow in the Open Science Grid is as follows:<ol><li>Stage input files to worker node.</li><li>Start processing…</li&...

2014

GPUs on the OSG

1 minute read

For a while, we have heard of the need for GPU’s on the Open Science Grid.  Luckily, HCC hosts numerous GPU resources that we are willing to share.  And wit...

OSG Usage Chrome App

less than 1 minute read

Hi, I’m Derek Weitzel.  You may remember me from such side projects as:<ul><li>OSG Android App</li><li>Redesign of Gratiaweb Interfac...

A HCC Dashboard with OSG Accounting

2 minute read

After the 2013 SuperComputing Conference, we found ourselves with a extra monitor at HCC.  Therefore, I set about creating a dashboard which can show the cur...

2013

OSG at Super Computing!

less than 1 minute read

Nebraska hosted a booth at the Super Computing conference this year.<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" sty...

Bosco @ CHEP

less than 1 minute read

This week, I attended the Computing in High Energy Physics (CHEP) 2013 conference in Amsterdam to present a poster on Bosco.<table align="center" cellpadd...

The NSA vs. my grid certificate

4 minute read

Note: Opinions or statements made in this post are solely my own, and are not the opinions of any organization that I work with or for.  Also, I probably got...

You asked, we delivered!

less than 1 minute read

A few users of BoscoR, and specifically GridR, have requested several major features:<ol><li>Allow custom packages to be installed on the remote...

How you know you are doing something right?

less than 1 minute read

I just received this from Dropbox:<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-righ...

CIC 13 Meeting

less than 1 minute read

This week, the Campus Infrastructures Community debuted the OSG Connect service.  Overall it was a very successful workshop.  I met 2 users that where very e...

Bosco & OSG @ XSEDE13

less than 1 minute read

Last week, several members of the Bosco team attended the XSEDE 13 Conference.  We enjoyed the great San Diego weather, and even caught a little ComicCon, wh...

Creating a native Mac Installer for Bosco

1 minute read

It has been our goal for some time to create a native installer for our supported platforms.  This week, we created a prototype for the first of those platfo...

The next step for Bosco: BoscoR

5 minute read

With the 1.1 release of Bosco in January, the Bosco team finally had a usable product for researchers.  It didn’t have the best usability, but it was a solid...

HCC OSG Workshop

less than 1 minute read

Over the last 2 days, HCC has been running a workshop on:<ol><li>Building an HTC Cluster</li><li>Running on a HTC Cluster</li>&...

Bosco at GPN Conference

less than 1 minute read

This week, I presented the Bosco poster at the Great Plains Network annual meeting.  It received a very good reception.  Many people where interested in wha...

Submitting R jobs with Bosco

1 minute read

The Bosco team has been working on integrating with the R statistics processing language.  We have chosen to modify the GridR package in order to integrate w...

Reprocessing CMS events with Bosco

1 minute read

Prior to the LHC long shutdown, the CMS experiment increased the trigger rate of the detector, therefore increasing the data coming off the detector.  The Ti...

Running Quantum Espresso on the OSG

1 minute read

While running Quantum Espresso on the Open Science Grid, we found a number of issues:<ul><li>OpenMPI needs to have an rsh binary.  Even if you ar...

Using Bosco to submit to Amazon EC2

4 minute read

A homework assignment for my storage class required running ~30 hours of benchmarks of the btrfs and ext4 filesystems.  I thought this would be an excellent ...

Rendering with Bosco

1 minute read

Example image rendered using HCC's distributed renderer At the Nebraska Holland Computing Center, we take pride in eating our own dog food.  So I want to hi...

Bosco 1.1.1 Release

less than 1 minute read

Today I am pleased to announce that Bosco version 1.1.1 has been released. This is a patch fix for the 1.1 to address 2 issues that where affecting users.  T...

Improving Gratia’s Web Interface

1 minute read

Over the winter break, I worked on improving the interface that most users use for OSG accounting.  When I returned from break, I worked with Ashu to integra...

Bosco 1.1 Release

less than 1 minute read

In the last few months, I have outlined the new features of Bosco 1.1.<ul><li>Mutli-Cluster Support</li><li>Multi-OS Support</li&...

2012

BOSCO v1.1 Features: Multi-Cluster Support

2 minute read

This is the forth in the in the series of features in the 1.1 release of BOSCO.  The previous posts have focused on SSH File Transfer, Single Port Usage, and...

BOSCO v1.1 Features: Multi-OS Support

1 minute read

This is part 3 of my ongoing series describing new features in BOSCO v1.1.  Part 1 covered file transfer over SSH, Part 2 covered single port usage.  This po...

BOSCO v1.1 Features: Single Port Usage

1 minute read

Welcome to part 2 of my ongoing series of v1.1 features for BOSCO.  Part 1 was on SSH File Transfer.This time, I’ll talk about a new feature that we didn’t p...

BOSCO v1.1 Features: SSH File Transfer

1 minute read

I am hoping to write about a few of the new features of BOSCO 1.1 before it comes out in December or January.  This is part 1 of that series:<h3>BOSCO ...

The Island Grid: Puerto Rico

3 minute read

In the last post, I discussed my trip to Puerto Rico.  Now that my trip is over, and I am back in Lincoln, I would like to share some of the successes of the...

Building a Campus Grid on a Island

less than 1 minute read

Hello everyone from San Juan, Puerto Rico. <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; ma...

CMS with the Campus Factory

2 minute read

The Campus Factory is usually used by small research groups to expand their available resources to those on the campus.  Of course, that’s not always easy f...

Day 4: Open Science Grid Summer School 2012

less than 1 minute read

Yesterday, I taught the class storage on the OSG.  Since I was teaching, I was unable to write or take any pictures.Today focuses on actual science on the OS...

Video of Igor’s Exercise

less than 1 minute read

Igor had a very interesting exercise during the OSG Summer School. It’s a little difficult to explain, but:Each student at the tables is a worker node.  The ...

Day 2 of OSG Summer School

less than 1 minute read

We had a great day yesterday at the OSG Summer School.  Not only was the weather great, but the exercises went very well (a testament to Alain). Monday...

Summer School Pictures

less than 1 minute read

I’m putting pictures from the summer school on my Google Plus.

Day 1 of OSG Summer School

less than 1 minute read

I was asked to teach at the OSG Summer School.  I think teaching the next generation of OSG users is a great opportunity.  This is how I learned about the OS...

Installing and Configuring glusterfs on EL6

2 minute read

I’m always interested in the newest technologies.  With the purchase of Gluster by RedHat, I figured it was time to give it a try.  We are always looking for...

Installing your own gratia webpage

1 minute read

In the OSG, we use a technology called Gratia for our accounting.  Every single job and data transfer on the OSG is accounted for in the database.  Of course...

In NY for CHEP

less than 1 minute read

Hanging out in Times Square I’m in New York this week for CHEP 2012.  I’ll be writing about my experiences here.  Stay Tuned.

Developments at Nebraska

1 minute read

I thought I would do a quick post about recent developments at Nebraska.Tusker & OSG<table align="center" cellpadding="0" cellspacing="0" class="tr-ca...

BOSCO + Campus Factory

1 minute read

Checklist while implementing CF + Bosco integration For the last several months, the campus infrastructure team has worked on software that will help users c...

OSG AHM 2012

2 minute read

This years all hands meeting was a great success!  There where a few sessions that I really enjoyed.AHM Pictures Campus CaucusThere where many user engageme...

Burning the LiveUSBs

less than 1 minute read

In my last post, I talked about the OSG LiveUSBs.  Now that the conference is next week, I have started burning the USB with the image. <table align="cent...

Building an OSG-Client LiveUSB

1 minute read

Nebraska/OSG USB keys to be distributed at OSG-AHM 2012 Since we have started using RPM’s for osg software, I’ve been interested if it where possible to make...

Ceph on Fedora 16

1 minute read

I’ve written before how to run ceph on Fedora 15, but now I’m working on Fedora 16.Last time I complained about how much ceph tries to do for you.  For bette...

Fedora 16 on OpenStack

2 minute read

After following Brian’s guide on installing Fedora 15 on OpenStack, I thought I would try my hand at Fedora 16.  There where a few differences.Filesystem Dif...

Initial EL6 Packages for OSG

less than 1 minute read

Last night I completed initial packages for EL6 support.  Just like for EL5, the first OSG component I created is the osg-wn-client.The osg-wn-client has a c...

2011

September Progress Report

4 minute read

Since September was a while ago, I’ll keep this short.  Most of Sept. was figuring out the class/work/research schedule.  It had been a 2 years since I’ve ta...

SC11 Visualization Prep

less than 1 minute read

Last year I put together a visualization for SuperComputing 10’ to show jobs moving over the OSG.<div class="separator" style="clear: both; text-align: ce...

CEPH on Fedora 15

2 minute read

Yesterday, I read a blog post using CEPH for a backend store for virtual machine images.  I’ve heard a lot about ceph in the last year, especially after it w...

OSG… There’s an app for that

1 minute read

So I thought this weekend would be a good time to learn something new.  I put together an OSG Android App.  Not all the features are working yet, but I tend ...

Upgrading the osg-client to Testing

less than 1 minute read

Another lively blog post about upgrading a set of packages to testing.  Last time it was the osg-wn-client, this time it’s the OSG-Client.  I will use a jira...

OSG Site Admin meeting: Day 2

less than 1 minute read

Another beautiful (read: HOT) day in Lubbock.<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" styl...

OSG Site Admin meeting: Day 1

less than 1 minute read

Hello from the hot, hot, hot Lubbock.<div class="separator" style="clear: both; text-align: center;"></div>Had a good morning of broad plenary ta...

Testing Release of OSG-Client RPM

less than 1 minute read

It’s finally here!  The first official testing release of the osg-client in RPM form.  We’re asking anyone and everyone to go out there and start testing.  T...

Job Visualization

less than 1 minute read

A year or so ago, Ian Stokes-Rees showed me a job visualization that he had put together to see how his workflows where going.  Gives a good overview per sit...

HCC Walltime Effeciency

less than 1 minute read

Rob’s post on walltime efficiency was interesting.  I did the same with HCC and found that we had much better efficiency than I expected.  Especially with G...

Need more cores, Scotty!

less than 1 minute read

A researcher/grid user from UNL has come back!  Monitoring 100K+ Jobs. Even though we have a lot of jobs idle, we are only able to get maybe 4.5k runni...

Gratia DB Rates

less than 1 minute read

In my recent discussions with NCSA on gratia, I found something that is probably obvious to other people, but I would still like to point out.<table align...

Using VOMS Groups and Roles

less than 1 minute read

I’ve been working on gratia graphs with Ashu Guru for the new GlideinWMS probe. I’ve stumbled upon very odd command line usage of voms-proxy-init. <p>I...

Overflowing the Tier 2

3 minute read

The last few days, the Nebraska CMS Tier 2 has been full. And I mean really full.<div style="text-align: center;"></div>The expected wait time f...

Day 7 at CERN - Saturday

2 minute read

For breakfast, fruit and a croissant.Brian and I walked to the bus stop at CERN, tried deciphering the bus schedule. We found the correct bus (after walking...

Day 6 at CERN - Friday

1 minute read

Breakfast was orange juice, fruit, and a croissant.Morning SessionThis morning was the site related meetings. We had a talk from Oliver from FNAL regarding ...

Day 5 at CERN - Thursday

1 minute read

Breakfast was Choco Puffs, Fruit, and Orange/Passion fruit juice.Today was largely Management meetings, so I was free to play tourist at CERN. I went to the...

Day 4 at CERN - Wednesday

1 minute read

Breakfast was Milk, Orange juice, Fruit, Vanilla (Natural) yogurt, and croissant.Morning SessionThis morning is Monitoring Task Force. This stuff is very in...

Day 3 at CERN - Tuesday

1 minute read

This morning was mostly spent fixing my user registration and getting my CERN account. Breakfast was a slice of keish, fruit, and an orange juice.Morning Se...

Day 2 At CERN

2 minute read

Woke up this morning at 7:30. Grabbed some breakfast across the street from the Hostel. I had orange juice with passion fruit (whatever that is), mixed fru...

Arriving at CERN

1 minute read

Had a decent flight into CERN. Aaron D. picked Brian and I up at the airport. The Hostel opens at 9:00, so we ate some breakfast in the cafeteria before we...

Making a new group in GlideinWMS VO Frontend

less than 1 minute read

The most difficult thing with the RPM (or the normal install) is making a new group in the GlideinWMS Frontend.The files you need to add are:<pre class="b...

Non-root RPM Install

less than 1 minute read

First, copy the database files from rpm. This is so you don’t have to reinstall things already installed on the system.cp /var/lib/rpm/* newprefix/To instal...

2010

Google Earth Fedora 13 Fixed!

less than 1 minute read

I was able to fix the install of google earth on fedora 13 by installing the i686 nvidia libraries.After adding rpm forge and installing the nvidia drivers (...

Condor Week

less than 1 minute read

Just got back from condor week. I gave a presentation on persistent jobs in Condor. Check out the other presentations:

2009

Hadoop Viz

less than 1 minute read

Something I’ve been working on.</param></param></param></embed>

Youtube Channel

less than 1 minute read

On my youtube channel, you can view the visualizations I’m working on.YouTube Channel</param></param></param></embed>

Adding Routes at startup

less than 1 minute read

To add routes to be started, the file located at /etc/sysconfig/network-scripts/ifup-routes will parse each file named route-eth*. The file should end up lo...

Senior Design Done

less than 1 minute read

It’s the end of the semester, and that means the end of Senior Design. If you are interested in a Linux 3D immersive game using Wii motes, here’s the link.

XWarpPointer

less than 1 minute read

Just came across a way to move the mouse pointer to an absolute position in X. Follow the code below:<pre class="code">int main(){ Display* xdi...

Parsing HTTP Responses

less than 1 minute read

Just so I don’t have to remember again, in order to seperate a header from the body of a http response, in C:<pre name="code" class="Bash">char* endOfH...

No devices: No Screens Detected.

less than 1 minute read

Had a problem today with the no screens detected on the new Viz wall computers. Solved it by first installing the Nvidia binaries straight from their websit...

Fuse DFS Error

less than 1 minute read

So, today I was having problems getting the fuse-dfs to work with my cluster. I was an error compiling fuse for my kernel (2.6.18-92.1.17.el5). It seems th...

Fish Monitoring

less than 1 minute read

After a few years of working on the Fish, I have finally published a video of them running, so that they could be demonstrated easier.</embed>

Viz Wall Computers

less than 1 minute read

Today I put together a computer spec for the viz wall we are building here at RCF.Here’s the link to the page:http://docs.google.com/Doc?id=dgjdk82m_23vp8rwfc2