Large Scale Data & Computing Resources

LIGO, Virgo, KAGRA and GEO data centers use tools from Open Science Grid (OSG) (OSG) to share their data and computing resources. Software for these examples are available in the IGWN Conda Distribution.

Data access

The Open Science Data Federation (OSDF) is a service provided by OSG that acts as a storage resource broker and optimizes data access through a worldwide network of data centers. Open data are stored under the '/gwdata' namespace. It can be accessed freely without any identification step. Similarly, LVK members can access proprietary data under '/igwn' namespace after having registered a SCiToken.

Data discovery

Data discovery consists in the identification of frame types and time segments. This can be done by using the open source python library GWDataFind .

OSDF access point servers:

• Open data discovery: datafind.gwosc.org
• Proprietary data: datafind.igwn.org

gw_data_find -r datafind.gwosc.org --show-types

Find available segments for the L1_GWOSC_O3a_16KHZ_R1 frame type from the Livingston observatory (L):

 gw_data_find -r datafind.gwosc.org -o L -t L1_GWOSC_O3a_16KHZ_R1 --show-times

The list of OSDF data file paths for the Handford observatory H1_LOSC_4_V1 frame type with times between GPS 1126259460 and 1126259464 is given by:

gw_data_find -r datafind.gwosc.org -o H -t H1_LOSC_4_V1 -s 1126259460 -e 1126259464 -u osdf

Data download

Once the wanted file OSDF path is known from data discovery, it can be downloaded directly on your computer using the pelican client. As an example, to download the data file from the H1 observatory with frame type H1_LOSC_4_V1 at GPS start time 1126256640:

pelican object get osdf:///gwdata/O1/strain.4k/frame.v1/H1/1126170624/H-H1_LOSC_4_V1-1126256640-4096.gwf my_local_copy.gwf

Python packages

Data can be queried and downloaded in a similar manner inside a python script using the packages gwdatafind and requests-pelican.

import requests_pelican as rp
from gwdatafind import find_urls
url = find_urls("H","H1_LOSC_4_V1",
1126259460, 1126259464, 
urltype='osdf', host="https://datafind.gw-openscience.org")
f = rp.get(url[0]).content

Computing Resources

Some researchers may want to make use of the OSG service OSPool to submit jobs to shared computing resources.

1. Create an account at osgconnect.net
2. Use the OSG Help Desk to find documentation, support, and office hours
3. Read the IGWN Guide to HTCondor to learn how to submit jobs to the OSPool using HTCondor