Large Scale Data & Computing Resources

LIGO, Virgo, KAGRA and GEO data centers use tools from Open Science Grid (OSG) (OSG) to share their data and computing resources. See here for more information.

Data access

The Open Science Data Federation (OSDF) is a service provided by OSG that acts as a storage resource broker and optimizes data access through a worldwide network of data centers. Open data are stored under the /gwdata namespace. It can be accessed freely without any identification step. Similarly, LVK members can access proprietary data under /igwn namespace after having registered a SCiToken.

OSDF access point servers:

• Open data discovery: datafind.gwosc.org
• Proprietary data: datafind.igwn.org

Software

To access data on the OSG, you need specialized software. We will use:

• GWDataFind: this is a command-line executable and a python library to query for the location of Gravitational-Wave Frame (GWF) files containing data from the LVK detectors.
• Pelican command-line client: this is the official command-line interface for OSDF.
• requests-pelican: a python library to provide simple interface to Pelican.

Examples

On the command-line

Data discovery consists in the identification of frame types and time segments. This is done with the gw_data_find command.

Available frame types among all detectors can be queried as:

gw_data_find -r datafind.gwosc.org --show-types

Find available segments for the L1_GWOSC_O3a_16KHZ_R1 frame type from the Livingston observatory (L):

gw_data_find -r datafind.gwosc.org -o L -t L1_GWOSC_O3a_16KHZ_R1 --show-times

The list of OSDF data file paths for the Handford observatory H1_LOSC_4_V1 frame type with times between GPS 1126259460 and 1126259464 is given by:

gw_data_find -r datafind.gwosc.org -o H -t H1_LOSC_4_V1 -s 1126259460 -e 1126259464 -u osdf

Once the wanted file OSDF path is known from data discovery, it can be downloaded directly on your computer using the pelican command. As an example, to download the data file from the H1 observatory with frame type H1_LOSC_4_V1 at GPS start time 1126256640:

pelican object get osdf:///gwdata/O1/strain.4k/frame.v1/H1/1126170624/H-H1_LOSC_4_V1-1126256640-4096.gwf my_local_copy.gwf

In Python

Data can be queried and downloaded in a similar manner inside a python script:

import os
import requests_pelican as rp
from gwdatafind import find_urls

url = find_urls( "H", "H1_LOSC_4_V1", 1126259460, 1126259464, urltype='osdf', host="https://datafind.gwosc.org")

with open(os.path.basename(url[0]), mode="wb") as file:
    file.write(rp.get(url[0]).content)

Computing Resources

Some researchers may want to make use of the OSG service OSPool to submit jobs to shared computing resources.

1. Create an account at osgconnect.net
2. Use the OSG Help Desk to find documentation, support, and office hours
3. Read the IGWN Guide to HTCondor to learn how to submit jobs to the OSPool using HTCondor