GeinQSAR Studio
Home › PubChem, files and other sources

PubChem, files and other sources

A ChEMBL publication for a 3D-QSAR series, PubChem assays and compound lists, your own measurements, and records downloaded before: how to find the data in each, and what to do when a service cannot be reached.

Whatever the source, the records arrive in one form and pass through the same curation, so the Study type, Curate, Review and Save pages work as described in Building a dataset from ChEMBL and Curation, review and saving. What differs is the second page, which takes the name of what it asks for.

ChEMBL publication

One paper is usually one chemical series measured in one laboratory with one protocol, which is what a 3D-QSAR study needs. The page is titled Publication.

  1. Enter the Publication id, the ChEMBL document identifier such as CHEMBL1135872, and press Look up. Upper or lower case does not matter. The card that appears gives the title, the journal with year, volume and first page, the authors and the DOI, so that you can confirm it is the paper you mean.
  2. Choose the Measurement type: IC50, Ki, Kd, EC50 or All types. With All types the most frequent potency type of the paper is used, and the Endpoint step reports which one.
  3. Optionally enter a Target id, for example CHEMBL220, to keep only the measurements of the paper on that target. Empty keeps every target the paper reports; if there are several, the Curate page asks which one to use (see the step Target).
  4. More options holds the Record limit.
The Publication page after Look up of chembl1135872: the card confirms the paper (Bioorg Med Chem Lett 2002, 12, 193, with its authors and DOI). IC50 is chosen and the target is restricted to CHEMBL220.
The Publication page after Look up of chembl1135872: the card confirms the paper (Bioorg Med Chem Lett 2002, 12, 193, with its authors and DOI). IC50 is chosen and the target is restricted to CHEMBL220.

Publication identifiers are found on the ChEMBL page of a paper, in the Publications column of a table curated from a ChEMBL target, and in the Sources tab of its Curate page, which lists the publications with the most compounds first. A good series has at least 20 compounds with exact values spanning three log units or more.

The study type 3D-QSAR congeneric series. The Assays line of the card ends in one publication, which is what the type adds to the regression settings.
The study type 3D-QSAR congeneric series. The Assays line of the card ends in one publication, which is what the type adds to the regression settings.
The curated series: 15 compounds from 40 records, after 25 censored values were removed. The findings say what a 3D study needs: a series of about 20 compounds at least, and no large gap in the activities.
The curated series: 15 compounds from 40 records, after 25 censored values were removed. The findings say what a 3D study needs: a series of about 20 compounds at least, and no large gap in the activities.

For a series larger and more varied than this, the Sources tab and the finding The set contains N different ring scaffolds tell you whether to restrict it to the most populated scaffold under 9 Series. A publication can also report the same compounds in several assays, one of which carries a placeholder value for every compound; restricting the set to one assay under 5 Assay then passes over the placeholder assay automatically.

PubChem BioAssay

The search box accepts three kinds of query. A number (or AID 1000) is read as the AID of one assay and loads it directly. A gene symbol such as ACHE or a protein accession such as P22303 lists the assays that test it, with their AID, name, method and the number of compounds and actives.

Listing the assays of a gene or a protein

A well-studied protein has thousands of assays, so the row under the search box says which of them are listed and in which order. The settings apply to the next search.

SettingWhat it doesWhy
Largest firstOn when the page opens. Lists the assays by the number of tested compounds. Off, PubChem's own order is used: by AID, the oldest first.The oldest assays are mostly tiny; in AID order the first 15 assays of ACHE have at most 24 compounds each.
With a potency valueLists only the assays that report a potency for their compounds (IC50, Ki, EC50, AC50).A regression needs such a value; an assay without one gives active and inactive only.
OrganismFor a gene symbol: the organism whose gene is meant, for example Homo sapiens. Empty lists the assays of every species that has a gene of that name.A gene symbol is not unique across species. A protein accession is specific already.
Smallest sizeLists only the assays that tested at least that many compounds. 0 lists all.Hides assays too small to model.

The line above the list is honest about the numbers: 25 of 1,199 assays (2,091 before the filters) means that 25 are listed, 1,199 pass the filters and PubChem holds 2,091. Show more lists the next window of the same ranking. A click on a column heading sorts the rows that are listed. A new search clears the selection. PubChem states a method for about three assays in five; the others are listed without one.

The assays of the gene symbol CHAT, largest first: 10 of 44 are listed and Show more adds the next ones. The listed rows are sorted by AID (the arrow on the heading). AID 51960 is selected and its summary is shown below the list.
The assays of the gene symbol CHAT, largest first: 10 of 44 are listed and Show more adds the next ones. The listed rows are sorted by AID (the arrow on the heading). AID 51960 is selected and its summary is shown below the list.

The chosen assay

Selecting an assay, or typing its AID, loads PubChem's own summary into a card: the name, the method and target, and the substances tested with their active, inactive and inconclusive counts. Value column is the readout that holds the activity value; the one the depositor marked as the active concentration is suggested and tagged Suggested until you confirm or change it. (none: outcomes only) downloads no value at all, for a classification by the depositor's outcome.

When the chosen readout was deposited without a unit, a field Unit of the values appears, set to (not stated), with the warning No unit was deposited: choose the unit the assay states. Take the unit from the assay description on PubChem. The pipeline never guesses a unit: without one, every value is removed at the step Units and p-scale.

AID 1000 typed into the search box: the card gives its name, the method (confirmatory, mevalonate kinase of Streptococcus pneumoniae) and 57 substances tested, 36 active and 21 inactive. The IC50 readout is suggested as the value column, and since it was deposited without a unit the page asks for one.
AID 1000 typed into the search box: the card gives its name, the method (confirmatory, mevalonate kinase of Streptococcus pneumoniae) and 57 substances tested, 36 active and 21 inactive. The IC50 readout is suggested as the value column, and since it was deposited without a unit the page asks for one.
What happens without the unit: 0 compounds from 57 records. The step Units and p-scale removed all 57 for No unit given, and the line under the funnel names that step as the place to change.
What happens without the unit: 0 compounds from 57 records. The step Units and p-scale removed all 57 for No unit given, and the line under the funnel names that step as the place to change.

Assays with rows on several targets

Some PubChem assays, mostly those deposited by BindingDB, hold results on several proteins in one table, one row per result: an assay named after a cholinesterase study can hold rows on acetylcholinesterase, on both monoamine oxidases and on a carboxylesterase. Results on different proteins are different experiments and are never pooled. For such an assay the page shows Target of the rows with every target of the table, its rows and its substances. Choose one, and only its rows are downloaded; nothing is chosen for you, and Next is refused with the number of targets until you choose. For a very large table the rows are counted on its first substances, and the tooltip says so.

AID 1796508, whose table holds rows on four targets. Target of the rows is set to acetylcholinesterase (P22303) with 93 rows and 93 substances; only those rows will be downloaded.
AID 1796508, whose table holds rows on four targets. Target of the rows is set to acetylcholinesterase (P22303) with 93 rows and 93 substances; only those rows will be downloaded.

More options

OptionChoicesWhen to use it
DownloadEvery outcome (default), Active only, Active and inactiveA screening assay can hold several hundred thousand substances, nearly all inactive. Active and inactive leaves out the inconclusive ones.
StructuresPubChem compound record (default), Depositor's SMILESThe compound record is the structure PubChem standardised. Use the depositor's SMILES when you need the structure exactly as deposited.
Record limit0 (every record)A first look at a screening assay. A limit takes the first substances in the table's order.

PubChem refuses a data table of more than 10,000 substances in one request, so a large assay is read in batches; the first 12,000 substances of a screening assay with 364,105 took 11 requests and 42 seconds including curation. A complete download of an assay of that size has not been measured: use the outcome filter and a record limit for such assays.

Classification by outcome

A PubChem assay can be labelled by its depositor's outcome rather than by a threshold: choose Classification and tick Labels from the depositor's outcome on the Study type page. Active and probe outcomes give active, inactive gives inactive, and a compound whose records say both is removed as conflicting. A compound whose outcome is inconclusive gets no class and is removed with Class cannot be determined.

AID 1000 in µM, classified at pIC50 6.50: 36 compounds, 21 active (58 %) and 15 inactive (42 %). The bar shows the split and the histogram marks the threshold. The finding warns that the inactive class holds 15 compounds, fewer than the 20 recommended.
AID 1000 in µM, classified at pIC50 6.50: 36 compounds, 21 active (58 %) and 15 inactive (42 %). The bar shows the split and the histogram marks the threshold. The finding warns that the inactive class holds 15 compounds, fewer than the 20 recommended.

PubChem compounds

This source looks up the structures of a list of compounds and gives a structure-only dataset, for a screening library to be predicted with a model you have built. The page is titled Compounds.

  1. Choose what The list holds: Names, CIDs, SMILES or InChIKeys. Nothing is pre-chosen.
  2. Enter the Identifiers, separated by semicolons. CIDs, SMILES and InChIKeys may also be separated by spaces or commas, so a pasted column of them works. Names may not, because a name can contain both (2,4-dinitrophenol, acetyl salicylic acid): read a column of names from a text file with Load list, one name per line.
  3. The table below shows the identifiers as they are read, with their count.
Two CIDs, 2244; 3672, read as two identifiers. The table lists them as the wizard parsed them, which is the check that the separators were understood.
Two CIDs, 2244; 3672, read as two identifiers. The table lists them as the wizard parsed them, which is the check that the separators were understood.

Only Screening library is offered on the Study type page, because the records hold no measurements. The curation then has two steps: Structure, which standardises each structure and keeps the largest component of a mixture, and Duplicates, which merges records with the same standardised structure. Identifiers PubChem does not know are named in the notes of the download, so that you can correct the list.

For a compound list the three activity study types are greyed and Screening library is chosen: structures only, salts stripped, neutralised, the largest component of a mixture kept.
For a compound list the three activity study types are greyed and Screening library is chosen: structures only, salts stripped, neutralised, the largest component of a mixture kept.
The curation of a library has two steps, Structure and Duplicates; here both compounds pass and no finding is raised.
The curation of a library has two steps, Structure and Duplicates; here both compounds pass and no finding is raised.

My own file

Your own table of structures and measured values passes through the same curation as the public data. The page is titled File, the Download page Read.

Choose the file with Browse, or type or paste the path and press Enter. The wizard reads the columns and the first rows, recognises the columns by their names (smiles, id, IC50 (nM), pKi, relation, unit, assay, reference) and tags the mapping Suggested until you change it. Check the mapping against the preview of the first rows.

A CSV file of four columns. The mapping was recognised from the header (tagged Suggested): SMILES as the structure, Code as the identifier, IC50 (nM) as the value in nM, activity type IC50, no relation column. The first rows of the file are shown below.
A CSV file of four columns. The mapping was recognised from the header (tagged Suggested): SMILES as the structure, Code as the identifier, IC50 (nM) as the value in nM, activity type IC50, no relation column. The first rows of the file are shown below.
FieldWhat it is
Structure columnThe column with the SMILES of each compound. For an SD file this is the connection table.
Identifier columnThe column that names each compound. With (none), the row number is used.
Value columnThe measured activity. Cells such as > 10, 12 nM, 1,5 or 1.2E-6 are understood: the relation and the unit are split off when the file has no column for them. Empty cells and placeholders (NA, n.d., -, inactive) give a record without a value, which the curation removes with its reason. With (none) the file is read as structures only.
Unit of the valuesThe unit when the file has no unit column: nM, uM, mM, M, pM, ug/mL, or p-scale, -log10(M) when the values already are pIC50 or pKi.
Activity typeWhat was measured, for example IC50, when the file has no type column. It names the response: IC50 gives pIC50.
Relation columnA column that holds =, < or > for each value. With (none) a relation written in the value cell is used, and a value without one is exact.
More columnsName column, Unit column (takes precedence over the unit above), Type column, Assay column, Reference column and Year column. The assay and reference columns let the Assay step and the Sources tab work as for public data.

A column the file does not have is refused by name. A file without a value column offers only Screening library; values in µg/mL need Convert mass concentrations under 3 Units.

The Read page of the file: 21 records. The notes say what reading found: one row without a numeric value, one value with its relation in the cell (> 10), one with its unit in the cell (12 nM), and no relation column.
The Read page of the file: 21 records. The notes say what reading found: one row without a numeric value, one value with its relation in the cell (> 10), one with its unit in the cell (12 nM), and no relation column.
The curated file: 17 compounds, pIC50 5.21 to 8.52. Relation removed the censored value and Units the row without a number; the findings report the small size and 15 activity cliffs.
The curated file: 17 compounds, pIC50 5.21 to 8.52. Relation removed the censored value and Units the row without a number; the findings report the small size and 15 activity cliffs.

Previous download

Every save writes a raw snapshot, name.raw.json, with the records exactly as they were retrieved. The source Previous download opens it and curates the same records again with other settings, without a network connection. The page is titled Snapshot; enter the path or use Browse. The release and the query of the original download come with the records, and the new name is suggested as name_recurated in the folder of the snapshot.

The snapshot of the regression dataset curated again as a classification at pIC50 5.5 with a grey zone of 0.1: 37 compounds, 7 active and 30 inactive. The dashed lines around the threshold mark the grey zone, and step 9 Class labels removed the 3 compounds inside it.
The snapshot of the regression dataset curated again as a classification at pIC50 5.5 with a grey zone of 0.1: 37 compounds, 7 active and 30 inactive. The dashed lines around the threshold mark the grey zone, and step 9 Class labels removed the 3 compounds inside it.

Use this to try a different study type, threshold or filter on data you have already examined. To regenerate the original dataset exactly, use replay instead.

Working offline and the cache

Answers of ChEMBL and PubChem are kept in a local cache in %APPDATA%\Geinforce\GeinQSAR\http-cache, so a repeated download is fast and the notes of the Download page say how many requests the cache answered. Activity pages are stored under the ChEMBL release they belong to, so pages of an older release are never mixed with those of the current one: after a new release the data are fetched again. Search results and counts are not tied to a release and are asked for again after 30 days.

The dependable way to work without a connection is the raw snapshot: Previous download and Data > Replay a provenance file never touch the network. A proxy that asks for a user name and password is not supported.

When something goes wrong

A request that fails shows the message of the service in red with Retry, and a help mark beside it that says what that kind of failure usually means. Nothing is left half done: a failed search lists nothing, and a failed download creates no dataset and is not repeated by itself.

A target search that could not be answered: the message is shown above the empty list with Retry, and the measurement types stay disabled.
A target search that could not be answered: the message is shown above the empty list with Retry, and the measurement types stay disabled.
A download that could not be made. The same message appears a second time at the foot of the page, as the reason why Next is refused.
A download that could not be made. The same message appears a second time at the foot of the page, as the reason why Next is refused.
MessageMeaningWhat to do
ChEMBL could not be reachedNo connection, or a proxy blocks it.Check the connection and press Retry. Records downloaded earlier are read from the cache.
ChEMBL did not answer in timeThe service is slow or busy.Retry in a moment.
The service reported an error or is limiting requests (help mark)A server error, or the service is throttling.Retry in a moment.
ChEMBL has no target CHEMBL123, PubChem has no assay AID ...The identifier does not exist.Check the spelling, or search by name.
ChEMBL was not contacted: Offline mode: this request is not in the local response cacheThe request was to be answered from the cache, and the cache does not hold it.Connect and retry, or work from a snapshot.
The source returned no recordsThe target has no measurement of the requested type.Choose another type.
No compound passed the curationThe message names the step that removed most and why.Change that step under Options; see Curation, review and saving.

Data sources and terms

ChEMBL is provided by EMBL-EBI under the CC BY-SA 3.0 licence; a dataset built from it must name the ChEMBL release, which the provenance file and the report record. PubChem is a public-domain resource of the U.S. National Library of Medicine; individual depositors may state their own terms. The builder identifies itself to both services as GeinQSAR/1.0 and stays below their request limits; when PubChem reports itself busy, requests are spaced 1.1 seconds apart.