Introduction to the CompTox Chemicals Dashboard (CCD)
Accessing chemical data is a vital step in many workflows related to chemical, biological, and environmental modeling. While there are many resources available from which one can pull data, the CompTox Chemicals Dashboard (CCD), built and maintained by the United States Environmental Protection Agency, is particularly well-designed and suitable for these purposes. Originally introduced in The CompTox Chemistry Dashboard: a community data resource for environmental chemistry, the CCD contains information on over 1.2 million chemicals as of December 2023.
The CCD includes chemical information from many different domains, including physicochemical, environmental fate and transport, exposure, usage, in vivo toxicity, and in vitro bioassay data. For information on data sources and current versions, please review the CCD Release Notes.
The CCD can be queried for one chemical at a time or using batch search.
Searching one chemical at a time (single-substance search)
In single-substance search, the user enters a full or partial chemical identifier (name, CASRN, InChiKey, or DSSTox ID) into a search box on the CCD homepage. Autocomplete can provide a list of possible matches. Figure 1 shows an example: the CCD landing page for the chemical Bisphenol A.
Figure 1: Bisphenol A CCD landing page
The different domains of data available for this chemical are shown by the tabs on the left side of the page: for example, “Physchem Prop.” (physico-chemical properties), “Env. Fate/Transport” (environmental fate and transport data), and “Hazard Data” (in vivo hazard and toxicity data), among others.
Batch search
In batch search, the user enters a list of search inputs, separated by new lines, into a search box. The user selects the type(s) of inputs by selecting one or more checkboxes – include chemical identifiers, monoisotopic masses, or molecular formulas. Then, the user selects “Display All Chemicals” to display the list of substances matching the batch-search inputs, or “Choose Export Options” to choose options for exporting the batch-search results as a spreadsheet. The exported spreadsheet may include data from most of the domains available on an individual substance’s CCD page.
Figure 2: CCD Batch Search
The user can download the selected information in various formats, such as Excel (.xlsx), comma-separated values (.csv), or different types of chemical table files (.e.g, MOL).
Figure 3: CCD Batch Search Customized Export
The web interface for batch search only allows input of 10,000 identifiers at a time. If a user wants to retrieve information for more than 10,000 chemicals, identifiers will need to be separated into multiple batches and searched separately.
Challenges of web-based dashboard search
Practicing researchers may follow a workflow that looks something like this:
- Start with a dataset that includes your chemical identifiers of interest. These may include chemical names, Chemical Abstract Service Registry Numbers (CASRNs), Distributed Searchable Structure-Toxicity Database (DSSTox) identifiers, or InChIKeys.
- Export the chemical identifiers to a spreadsheet. Often, this is done by importing the data into an environment such as R or Python, in order to do some data wrangling (e.g., to select only the unique substance identfiers; to clean up improperly-formatted CASRNs; etc.). Then, the identifiers are saved in a spreadsheet (an Excel, .csv, or .txt file), one chemical identifier per row.
- Copy and paste the chemical identifiers from the spreadsheet into the CCD Batch Search box. If there are more than 10,000 total chemical identifiers, divide them into batches of 10,000 or less, and search each batch separately.
- Choose your desired export options on the CCD Batch Search page.
- Download the exported spreadsheet of CCD data. By default, the downloaded spreadsheet will be given a file name that includes the timestamp of the download.
- Repeat steps 3-5 for each batch of 10,000 identifiers produced in step 2.
- Import the downloaded spreadsheet(s) of CCD data into the analysis tool you are using (e.g. R or Python).
- Merge the table(s) of downloaded CCD data with your original dataset of interest.
- Proceed with research-related data analysis using the chemical data downloaded from the CCD (e.g., statistical modeling, visualization, etc.)
Because each of these workflow steps requires manual interaction with the search and download process, the risk of human error inevitably creeps in. Here are a few real-world possibilities (the authors can neither confirm nor deny that they have personally committed any of these errors):
- Researchers could copy/paste the wrong identifiers into the CCD batch search, especially if more than 10,000 identifiers need to be divided into separate batches.
- Chemical identifiers could be corrupted during the process of
exporting to a spreadsheet.
- If a researcher opens and resaves a CSV file using Microsoft Excel, any information that appears to be in date-like format will be automatically converted to a date (unless the researcher has the most recently-updated version of Excel and has selected the option in Settings that will stop Excel from auto-detecting dates). This behavior has long been identified as a problem in genomics, where gene names can appear date-like to Excel (Abeysooriya et al. 2021). It also affects cheminformatics, where chemical identifiers can appear date-like to Excel. For example, the valid CASRN “1990-07-4” would automatically be converted to “07/04/1990” (if Excel is set to use MM/DD/YYYY date formats). CCD batch search cannot recognize “07/04/1990” as a valid chemical identifier and will be unable to return any chemical data.
- Researchers could accidentally rename a downloaded CCD data file to overwrite a previous download, e.g. when searching multiple batches of identifiers.
- Researchers could mistakenly import the wrong CCD download file back into their analysis environment e.g. when searching multiple batches of identifiers.
Moreover, the manual stages of this kind of workflow are also non-transparent and not easily reproducible. Utilizing APIs for data exploration and retrieval can alleviate these concerns.
Application Programming Interfaces (APIs) for Automated Batch Search of the CCD
Recently, the Center for Computational Toxicology and Exposure (CCTE) developed a set of Application Programming Interfaces (APIs) that allows programmatic access to the CCD, bypassing the manual steps of the web-based batch search workflow. APIs effectively automate the process of accessing and downloading the data that populates the CCD.
The CCTE APIs are publicly available at no cost to the user. However, in order to use the CCTE APIs, users must have a individual API key. The API key uniquely identifies the user to the CCD servers and verifies that you have permission to access the database. Getting an API key is free, but requires contacting the API support team at ccte_api@epa.gov.
The APIs are organized into sets of “endpoints” by data domains:
Chemical, Hazard, and
Bioactivity. A view from the Chemical APIs web
interface is pictured below.
Figure 4: CCTE API Chemical Endpoints
On the left side of each domain’s web interface page, there will be
several different tabs listed depending on information requests
available within the domain. In Figure 4, the
Chemical Details Resource endpoint provides basic chemical
information; the Chemical Property Resource endpoint
provides more comprehensive physico-chemical property information; the
Chemical Fate Resource endpoint provides chemical fate and
transport information; and so on.
Authentication
Authentication, found in upper left tab on each web
interface page, is required to use the APIs. To authenticate themselves
in the API web interface, the user must input their unique API key.
Figure 5: API Key Authentication
Request Construction
APIs effectively automate the process of accessing and downloading the data that populates the CCD. APIs do this via requests using the Hypertext Transfer Protocol (HTTP) that enables communication between clients (e.g. your computer) and servers (e.g. the CCD).
In the CCTE API web interface, the colored boxes next to each endpoint indicate the type of the associated HTTP method. GET is used to request data from a specific web resource (e.g. a specific URL); POST is used to send data to a server to create or update a web resource. For the CCTE APIs, POST requests are used to perform multiple (batch) searches in a single API call; GET requests are used for non-batch searches.
You do not need to understand the details of POST and GET requests in
order to use the API. Let’s consider constructing an API request to
Get data by dtxsid under the
Chemical Details Resource.
Figure 6: Get Details by DTXSID
The web interface has two subheadings:
- Path Parameters contain user-specified parameters that are required in order to tell the API what URL (web address) to access. In this case, the required parameter is a string for the DTXSID identifying the chemical to be searched.
- Query-String Parameters contain user-specific
parameters (usually optional) that tell the API what specific type(s) of
information to download from the specified URL. In this case, the
optional parameter is a
projectionparameter, a string that can take one of five values (chemicaldetailall,chemicaldetailstandard,chemicalidentifier,chemicalstructure,ntatoolkit). Depending on the value of this string, the API can return different sets of information about the chemical. If theprojectionparameter is left blank, then a default set of chemical information is returned.
The default return format is displayed below and includes a variety of fields with data types represented.
Figure 7: Get Details by DTXSID, Return Format
Pictured below is an example of returned Details for Bisphenol A with
the chemicaldetailstandard value for
projection selected.
Figure 8: Returned Details for Bisphenol A
Introduction to ccdR
Formatting an http request is not necessarily intuitive nor worth the time for someone not already familiar with the process, so these endpoints may provide a resource that for many would require a significant investment in time and energy to learn how to use. However, there is a solution to this in the form of the R package ccdR.
ccdR was developed to streamline the process of accessing the information available through the CCTE APIs without requiring prior knowledge of how to use APIs.
Package Settings
Users can run library(ccdR) to install from CRAN or
install the development version of ccdR like so:
if (!library(devtools, logical.return = TRUE)){
install.packages(devtools)
library(devtools)}
devtools::install_github("USEPA/ccdR")API Key Storage
As previously described, a user must have an API key to use in order
to access the CCTE APIs. This can be obtained from the admins of the
CCTE APIs by emailing CCTE API
Admins. In the example code, the API key will be stored as the
variable my_key.
my_key <- 'YOUR_CCTE_API_key'For general use of the package, the user may use the function
register_ccdr() to store the API key in the current session
or more permanently for access across sessions.
# This stores the key in the current session
register_ccdr(key = '<YOUR API KEY>')
# This stores the key across multiple sessions and only needs to be run once. If the key changes, rerun this with the new key.
register_ccdr(key = '<YOUR API KEY>', write = TRUE)Once the API key is stored, the default display setting is turned off for protection. To change this, use the following functions as demonstrated.
# To show the API key
ccdr_show_api_key()
getOption('ccdr')$display_api_key
# To hide the API key
ccdr_hide_api_key()
getOption('ccdr')$display_api_keyFinally, to access the key, use the ccte_key()
function.
ccte_key()Quick Start Examples
As some quick start examples, we demonstrate the ease* of retrieving the information across endpoints for Bisphenol A using ccdR. *This is in contrast to the approach using the CCD or API web interface.
For additional examples and more comprehensive documentation on each endpoint, consider reviewing the other ccdR vignettes for the data domain of interest.
Chemical APIs
In this section, several ccdR functions are used to access different types of information from the CCTE Chemical APIs.
Chemical Details Resource
The function get_chemical_details() takes in either the
DTXSID or DTXCID of a chemical and the user-specific API key. Relevant
chemical details for Bisphenol A, which has DTXSID “DTXSID7020182”, are
obtained in a data.table. output below:
bpa_details <- get_chemical_details(DTXSID = 'DTXSID7020182',
API_key = my_key)
bpa_details <- data.table::as.data.table(bpa_details)
head(bpa_details)Chemical Property Resource
The function get_chem_info() returns phys-chem
properties for the selected chemical, and can be filtered to
‘experimental’ or ‘predicted’ if desired.
Here all phys-chem properties are returned for Bisphenol A:
bpa_info <- get_chem_info(DTXSID = "DTXSID7020182",
API_key = my_key)
bpa_info <- data.table::as.data.table(bpa_info)
head(bpa_info)Request can be filtered to return experimental results only.
bpa_info_experimental <- get_chem_info(DTXSID = "DTXSID7020182",
type = 'experimental',
API_key = my_key)
bpa_info_experimental <- data.table::as.data.table(bpa_info_experimental)
head(bpa_info_experimental)Hazard APIs
In this section, several ccdR functions are used to access different types of information from the CCTE Hazard APIs.
Hazard Resource
The function get_hazard_by_dtxsid() retrieves hazard
data (all human or ecological toxicity data) for a given chemical based
on input DTXSID. get_human_hazard_by_dtxsid() and
get_ecotox_hazard_by_dtxsid() can filter returned hazard
results for the given chemical to human or ecological toxicity data,
respectively.
Here all hazard data is returned for Bisphenol A:
bpa_hazard <- get_hazard_by_dtxsid(DTXSID = 'DTXSID7020182',
API_key = my_key)
bpa_hazard <- data.table::as.data.table(bpa_hazard)
head(bpa_hazard)Request can be refined to return results for human hazard,
bpa_human_hazard <- get_human_hazard_by_dtxsid(DTXSID = 'DTXSID7020182',
API_key = my_key)
bpa_human_hazard <- data.table::as.data.table(bpa_human_hazard)
head(bpa_human_hazard)or EcoTox results.
bpa_eco_hazard <- get_ecotox_hazard_by_dtxsid(DTXSID = 'DTXSID7020182',
API_key = my_key)
bpa_eco_hazard <- data.table::as.data.table(bpa_eco_hazard)
head(bpa_eco_hazard)Bioactivity APIs
In this section, several ccdR functions are used to access different types of information from the CCTE Bioactivity APIs.
Bioactivity Resource
The function get_bioactivity_details() retrieves all
bioactivity data for a given chemical based on input DTXSID.
bpa_bioactivity <- get_bioactivity_details(DTXSID = 'DTXSID7020182',
API_key = my_key)
bpa_bioactivity <- data.table::as.data.table(bpa_bioactivity)
head(bpa_bioactivity)The function get_bioactivity_details() can also be used
to retrieve all bioactivity data for a given endpoint, based on input
AEID (assay endpoint identifier).
assay_id_search <- get_bioactivity_details(AEID = 42,
API_key = my_key)
assay_id_search <- data.table::as.data.table(assay_id_search)
head(assay_id_search)Conclusion
The ccdR package provides a streamlined approach to accessing data from the CCD for users with little or no prior experience using APIs.
For additional examples and more comprehensive documentation on each endpoint, consider reviewing the other ccdR vignettes for the data domain of interest.