3. Similarity Searching

Similarity Searching is the process of aligning the sequences at this stage of the pipeline (after filtering and refinement has been performed) to reference databases. These databases often come from NCBI RefSeq (https://www.ncbi.nlm.nih.gov/refseq/ or UniProt Swiss-Prot (https://www.uniprot.org/). The assumption here is that the user has already configured some of these databases for use with DIAMOND. If not, these can be done during the Configuration stage of EnTAP, refer back to that page for more information.

DIAMOND (https://github.com/bbuchfink/diamond) is utilized for Similarity Searching over traditional methods, such as BLAST, due to its improved speeds and accuracy.

3.1. Running Similarity Searching

Similarity Searching is a mandatory stage of EnTAP. In order to run this, the user must input multiple DIAMOND configured databases (should have the ‘.dmnd’ extension) through the database command. There are many customizable parameters for this stage as well that can be seen in the table below. An important piece to consider, is that EnTAP will utilize a unique method of best alignment selection that selects the best alignment per database utilizing a variety of parameters. This also factors in contaminant filtering!

Note

It is highly recommended to use a UniProt database so EnTAP can pull additional alignment information

3.2. Similarity Search Commands

Similarity Search Flags

param

description

location (cmd/R-ini,E-ini)

qualifier

example

database / d

Specify up to 5 DIAMOND indexed (.dmnd) databases to run similarity search against

R-ini

multi-string

/path/to/diamond/database.dmnd

data-type
Specify which EnTAP database you’d like to use for execution

    1. Binary Database (default) - This will be much quicker and is recommended

    1. SQL Database - Slower although will be more easily compatible with every system

R-ini

integer

0

contam / c

Specify contaminants to be used during Simililarity Search best hit selection. Contaminants can be selected by species or through a specific taxon (insecta) from the NCBI Taxonomy Database. If your taxon is more than one word just replace the spaces with underscores (_). Alignments will be flagged as contaminants and will be lower scoring compared to other alignments.

R-ini

multi-string

insecta

taxon

This flag will allow for ‘taxonomic favoring’ of hits that are closer to your target species or lineage. Any lineage can be used as referenced by the NCBI Taxonomic database, such as genus, phylum, or species. Format must replace all spaces with underscores (‘_’)

R-ini

string

homo_sapiens

e

Specify E-value cutoff for Similarity Searching results (in scientific notation format).

R-ini

scientific

10E-5

tcoverage

Specify minimum target coverage for similarity searching

R-ini

float

50

qcoverage

Specify minimum query coverage for similarity searching

R-ini

float

50

uninformative

Comma-deliminated list of terms you would like to be deemed “uninformative”. Any alignments during Similarity Searching tagged as uninformative will be scored lower

R-ini

string

conserved, predicted, unnamed, hypothetical, putative, unidentified, uncharacterized, unknown, uncultured, uninformative

diamond-exe

Specify the execution method for DIAMOND. This can be a path to the diamond file generated during installation, or simply the command if installed globally

E-ini

string

diamond
Similarity Search - DIAMOND Specific Flags

param

description

location (cmd/R-ini,E-ini)

qualifier

example

diamond-sensitivity

Specify the DIAMOND sensitivity used against input DIAMOND databases (Similarity Searching and HGT Analysis). Sensitivities are based off of DIAMOND documentation with a higher sensitivity generally taking longer but giving a higher alignment rate. Sensitivity options are fast, mid-sensitive, sensitive, more-sensitive, very-sensitive, ultra-sensitive.

R-ini

string

very-sensitive
NCBI API Flags

param

description

location (cmd/R-ini,E-ini)

qualifier

example

ncbi-api-key

Enter your personal NCBI API key, if available. This can be assigned to you through your NCBI account. Although not required, enabling EnTAP to use your API key will allow for much quicker accessions to the NCBI database. Your API key will only be used to access the NCBI database and only stored locally. If you are a contributor to EnTAP, DO NOT accidentally commit your API key to git!

R-ini

string

n/a

ncbi-api-enable

Allow EnTAP to access the NCBI database API to pull additional information for your data during Similarity Searching if searching against a NCBI database.

R-ini

bool

true

3.3. EnTAP Best Alignment Selection

EnTAP incorporates a unique method of selecting the best alignment per database, then overall. This utilizes parameters such as E-Value, Coverage, Contaminant Filtering, and Taxonomic Favoring.

Best Alignment Selection For Each Database:

  1. Examine E-Value

    • If E-Value difference is high, select the smallest E-Value alignment

    • If E-Value difference is low, continue

  2. Examine Coverage

    • If Coverage difference is high, select the larger Coverage alignment

    • If Coverage difference is low, continue

  3. Contaminant Filtering

    • If one alignment is a contaminant, select non-contaminant

    • If both/neither alignments are contaminants, continue

  4. Taxonomic Favoring and Informativeness

    • Select alignment that is closer in lineage to our target species and more informative

3.3.1. Contaminant Filtering

Contaminant Filtering leverages the NCBI Taxonomic Database (https://www.ncbi.nlm.nih.gov/taxonomy) to tag alignments that could be considered a contaminant. Contaminants can be introduced during collection or processing of a sample. A contaminant is essentially a species that is not of the target species you are collecting. Some common contaminants are bacteria and fungi that can sometimes be found within collected samples. Oftentimes, researchers would like to remove these sequences from the dataset.

In order to use Contaminant Filtering, the user must use the contam flag to select multiple (comma-separated) species/taxon from the NCBI Taxonomy Database. When inputting these, any spaces in the taxon must be replaced by an underscore (‘_’). Any alignments containing the contaminant taxon will be flagged as such and unfavored during best alignment selection.

As an example with common contaminants within the entap_run.params file:

contam=insecta,fungi,bacteria

Note

Sometimes the best alignment can be a contaminant! EnTAP will flag this and allow the user to decide whether or not they would like to retain it

3.3.2. Taxonomic Favoring

During best alignment selection of similarity searched results, taxonomic consideration can utilized. If a certain lineage (pinus) is specified, hits closer in taxonomic lineage to this selection will be chosen. Any lineage such as species/kingdom/phylum can be utilized as long as it is contained within the NCBI Taxonomic Database. If it is not located within the database, EnTAP will stop the execution immediately and let you know!

This feature can be utilized via the entap_run.params file. An example can be seen below (remember to replace any spaces with an underscore):

taxon=pinus_taeda

3.3.3. Informativeness

Informativeness is another metric that is used during selection of the best alignment. Oftentimes reference databases may have terms such as ‘unknown’ in the descriptions of alignments where certain information may not be known about that alignment. EnTAP will attempt to select alignments that are well established rather than these ‘uninformative’ alignments.

Any term can be used via the uninformative flag in the entap_run.params, so you are not limited! Below are defaults used by EnTAP as comma-separated:

uninformative=conserved, predicted, unnamed, hypothetical, putative, unidentified, uncharacterized, unknown, uncultured, uninformative

3.4. Interpreting the Results

The /similarity_search/DIAMOND directory will contain all of the relevant information for the Similarity Searching stage of the pipeline. This folder will contain the files generated from DIAMOND as well as files generated from EnTAP. Files generated from EnTAP for each individual reference database are contained within the /similarity_search/DIAMOND/processed/database_ref, while the overall analysis compiling the results of each reference database are contained within the /similarity_search/DIAMOND/overall_results.

The same files are repeated across databases and across the overall results, so I will only go into detail for each file once below with an input transcriptome labelled ‘species’ and a reference database labelled ‘ref_database’.

Similarity Search Results

filename

description

directory

blastp_species_ref_database.out

Generated from DIAMOND. Contains a lot of information from the DIAMOND search including e-value, coverage, reference database descriptions, and much more. This is a typical output file from a BLAST type of search. The filename is prepended with either blastp or blastx depending on if the input sequence is protein or nucleotide. A file like this will be generated for each reference database used.

/similarity_search/DIAMOND

blastp_species_ref_database_std.err/.out

Generated from DIAMOND. These files are will contain any error or general information produced from the DIAMOND run.

/similarity_search/DIAMOND

diamond_annotated.faa/.fnn/.tsv

Generated from EnTAP. Contains all of the best alignments (protein/nucleotide format) that were selected from this database, or overall combining the results from each database used. Since this contains all best alignments, it may contain contaminants or uninformative alignments. Sometimes a contaminant can be the best alignment! Note: Protein or nucleotide information may not be available to report depending on if the input sequence is protein or nucleotide.

/similarity_search/DIAMOND/processed/database_ref or /similarity_search/DIAMOND/overall_results

diamond_annotated_contam.faa/.fnn/.tsv

Generated from EnTAP. Contains all of the transcripts flagged as contaminants (protein/nucleotide format) that are a subset of the diamond_annotated best alignment files. Again this will be seen per database, or overall combining the results from each database used.

/similarity_search/DIAMOND/processed/database_ref or /similarity_search/DIAMOND/overall_results

diamond_annotated_without_contam.faa/.fnn/.tsv

Generated from EnTAP. Contains all of the transcripts NOT flagged as contaminants (protein/nucleotide format) that are a subset of the diamond_annotated best alignment files. With this in mind: best_hits = best_hits_no_contam + best_hits_contam. Again this will be seen per database, or overall combining the results from each database used. These sequences are separated from the rest for convenience if you would like to examine them differently

/similarity_search/DIAMOND/processed/database_ref or /similarity_search/DIAMOND/overall_results

unannotated.faa/.fnn/.tsv

Generated from EnTAP. Sequences (protein/nucleotide) from the transcriptome that did not hit against this particular reference database. This does not include sequences that were lost during expression filtering or frame selection. Again this will be seen per database, or overall combining the results from each database used.

/similarity_search/DIAMOND/processed/database_ref or /similarity_search/DIAMOND/overall_results

diamond_unselected_hits.faa/.fnn/.tsv

Generated from EnTAP. Similarity searching can result in several hits for each query sequence. With only one best alignment being selected, the rest are unselected and end up here. Unselected hits can be due to a low e-value, coverage, or other properties EnTAP takes into account when selecting hits

/similarity_search/DIAMOND/processed/database_ref or /similarity_search/DIAMOND/overall_results

3.4.1. Similarity Search Headers

TSV files generated from EnTAP will have the following headers for Similarity Searching (from left to right). Other headers may be present from previous stages of EnTAP (such as Frame Selection or Expression Filtering).

  • SeqSearch_Subject_Sequence

  • SeqSearch_Percent_Identical

  • SeqSearch_Alignment_Length

  • SeqSearch_Mismatches

  • SeqSearch_Gap_Openings

  • SeqSearch_Query_Start

  • SeqSearch_Query_End

  • SeqSearch_Subject_Start

  • SeqSearch_Subject_End

  • SeqSearch_E_Value

  • SeqSearch_Coverage

  • SeqSearch_Description

  • SeqSearch_Species

  • SeqSearch_Taxonomic_Lineage

  • SeqSearch_Origin_Database

  • SeqSearch_Contaminant

  • SeqSearch_Informative

If you ran Similarity Searching against a UniProt database, EnTAP will pull additional UniProt information for your alignments. The following headers will then be added to your output TSV file:

  • Database_UniProt_InterProScan

  • Database_UniProt_OrthoDB

  • Database_UniProt_KEGG _Terms

  • Database_UniProt_Protein_Domains

  • Database_UniProt_GO_Biological

  • Database_UniProt_GO_Cellular

  • Database_UniProt_GO_Molecular

If you ran Similarity Searching against a NCBI database, EnTAP will pull additional information for your alignments. The following headers will then be added to your output TSV file:

  • SeqSearch_NCBI_GeneID