Use clusters in spatial conditions

Spatial conditions are a core part of Biodiverse

Most people seem to focus on using single cells for their analysis and trying to find the ideal cell size.  This is missing much of the benefit of spatial analyses.  You are not constrained to using single cells in isolation.  

You can analyse regions around each focal location (processing group) using geometric shapes like circles.  Varying the size of the window gives an understanding of the spatial scale of the patterns (the operational scale).  However, there is no need to be geometric - you can use arbitrarily complex spatial conditions based on polygon features, proximity and/or matching text.  See for example Laffan and Crisp (2003) and Laity et al. (2015).  

You can also use cluster (and region grower) analyses to define your spatial windows.  These allow you to let the data define the regions, with the calculations then applied giving you more understanding of the groupings that have been identified.  Care needs to be taken with interpretation due to the risk of circularity, but that's not unusual.  And sometimes you just want to understand something about the assemblage that falls under a node (branch).  You might also be interested in the environmental properties associated with a cluster.

One issue with the cluster approach is that it can be difficult to use the branches in a spatial condition for a different analysis.  Consider the case where one wants to spatially partition a randomisation so labels are kept within their associated clusters (for a given cluster cutoff).  You could export the clusters to shapefile format, extract the relevant features to a new shapefile, and then use that in a new spatial condition.  But that's a lot of work and not easy for people less familiar with geoprocessing and GIS.  

From version 4 you can access the set of groups under a cluster analysis and use that to define spatial conditions (actually it is in the 3.99_003 development version).  This can use any of the current cutting methods, so you can slice by distance from the tips, depth, or number of clusters from the root using the sp_points_in_same_cluster condition.  You can also select individual branches (nodes) by name (sp_point_in_cluster).   

Some snippets are below that can be copied into your spatial conditions windows.  No screenshots this time, but I can add a new post of that is needed.  

Note that the cluster analysis being referred to must be in the same basedata.  

## sp_points_in_same_cluster examples

#  Try to use the highest four clusters from the root.
#  Note that the next highest number will be used
#  if four is not possible, e.g. there might be five
#  siblings below the root.  Fewer will be returned
#  if the tree has insufficient tips.
sp_points_in_same_cluster (
  output       => "some_cluster_output",
  num_clusters => 4,
)

#  Cut the tree at a distance of 0.25 from the tips
sp_points_in_same_cluster (
  output          => "some_cluster_output",
  target_distance => 0.25,
)

#  Cut the tree at a depth of 3 from the root.
#  The root is depth 1.
sp_points_in_same_cluster (
  output          => "some_cluster_output",
  target_distance => 3,
  group_by_depth  => 1,
)

#  Select four clusters below a specified node
sp_points_in_same_cluster (
  output       => "some_cluster_output",
  num_clusters => 4,
  from_node    => '118___',  #  use the node's name
)

#  target_distance is ignored if num_clusters is set

#  so this is the same as the first example
sp_points_in_same_cluster (
  output          => "some_cluster_output",
  num_clusters    => 4,
  target_distance => 0.25,
)


## sp_point_in_cluster examples

#  This will select any element that is a terminal in the cluster output
#  It is useful when the cluster analysis was run under
#  a definition query to reduce the number of elements clustered,

#  and you want the same set of elements.
sp_point_in_cluster (
  output       => "some_cluster_output",
)

#  Now specify a cluster within the output
sp_point_in_cluster (
  output       => "some_cluster_output",
  from_node    => '118___',  #  use the node's name
)

#  Specify an element to check instead of the current
#  processing element.
sp_point_in_cluster (
  output       => "some_cluster_output",
  from_node    => '118___',  #  use the node's name
  element      => '123:456', #  specify an element to check
)

 

Shawn Laffan

02-May-2022

For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse/  

To see what else Biodiverse has been used for, see https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList 

You can also join the Biodiverse-users mailing list at https://groups.google.com/group/Biodiverse-users 

 

Updated handling of Cluster and Region Grower analyses in randomisations

Randomisations in Biodiverse are used to assess the statistical significance of a set of analysis results given some randomisation scheme such as shuffling the species around the map (labels across groups in Biodiverse terms), subject to constraints such as each cell (group) must maintain the same number of species. Randomisations are key to interpreting where results differ from what would be expected, and are integral to protocols such as the Categorical Analysis of Paleo and Neo Endemism (CANAPE). 

The basic process of the randomisation is this.  For each iteration of the randomisation analysis, Biodiverse will:

1. Create a new basedata object with a random allocation of labels to groups
2. For each analysis in the basedata 
1. Regenerate a version using the randomised basedata
2. Compare the values of the analyses from the original and randomised basedata and track if they are higher or lower, on a cell by cell basis
3. Track basic statistics of the distribution to allow the calculation of the mean and standard deviation, and thus z-scores (this is new in Version 4 - there will be more on this in another post).

The tracking is used to reduce memory usage, as the randomised basedatas and outputs can be discarded as soon as they have been collated.  This is more efficient than keeping all the results in memory for later ranking, especially with large data sets such as those used for Mishler et al. (2020).

A huge amount of work has been spent over the last several years to make the randomisation process go faster and scale better with larger data sets, with recent versions being orders of magnitude faster than some of the earlier versions. In some cases analyses now take minutes where they used to take days. However, one slow-down that still affects things in version 3.1 is the effect of tree structures in a basedata. These are the Cluster and Region Grower analyses. 

The reason the Cluster and Region Grower analyses slow things down is simply that they take longer to calculate.  A spatial analysis only need to pass over each processing group (cell) once, so one can think of it as N calculations.  In comparison, a cluster analysis will compare each group with each other group to create its matrix.  If one is clustering a full data set then this will be N(N-1)/2 calculations, but the same scaling effect applies for subsets defined using definition queries (e.g. for CANAPE).  Even when one considers that the spatial analysis might include many neighbours for each group, the Cluster and Region Grower analyses take much longer.  And then the Cluster analysis needs to process the matrix.

Even with the slow-down, a comparison of the randomised Cluster or Region Grower analysis with the original is not always informative.  

In Biodiverse Version 4, this process has been changed.  Cluster and Region grower analyses are now skipped by default.  This will substantially speed up randomisations containing such analyses.  

If you still want to run them then there is an option to do so.  See the red arrow in the screenshot below.  

Randomisations of Cluster and Regions Grower analyses is off by default in Version 4, but can be re-enabled if needed (see red arrow).

One point to note is that any calculations per node (branch) are still done.  This was modified some time ago to use the set of randomised groups under each branch from the original, non-randomised tree.  This works because the group IDs are the same across both the original and the randomised basedatas.  It is just that the randomised version has a randomly allocated set of labels. 

Shawn Laffan

06-Nov-2020

For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse/ 

For the full list of changes in the 3.99 development series, leading to version 4, see https://github.com/shawnlaffan/biodiverse/wiki/ReleaseNotes#version-399-dev-series 

To see what else Biodiverse has been used for, see https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList 

You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users 

Export cluster analyses to shapefiles

In Version 3 of Biodiverse it is possible to export your cluster and region grower analyses to shapefile formats.

This makes it easier to use the regions in geospatial analyses, for example as conditions in spatial analyses in Biodiverse.  Perhaps the best example is that you can generate a regionalisation using one data set, and then assess its values for one or more other data sets.  See for example González-Orozco et al. (2013, 2014a and 2014b).

The option is available via the export menu for cluster and region grower analyses.  The options are all the usual ones, although that is not many in this case.

The option is chosen from the export menu for a cluster (or region grower) analysis.  

There are not that many choices, but Shape type can be exported as POLYGON or POINT geometries (shapes).  You can attach lists of results, or choose to only export the geometries themselves.

You might need to run a selection to highlight the branch you are interested in.  This example is branch 120___, which corresponds to the red cluster in the first screenshot above.  You might also choose to use a Definition Query (ArcGIS) or Layer Subset (QGIS) to filter out the geometries you are not interested in.  

And here the component polygons that form the polygons for branch 120___ are highlighted in red.

There are three points to watch for.

1.  If you export a list with the data, then each polygon or point feature is repeated for each item in the list.  This is due to shapefile field names being limited to 11 characters, which is far too short for many of the list entries in Biodiverse.  (Support for the GeoPackage format is in the works to obviate this issue).

2.  The exported features (geometries) are multipolygon or multipoint.  This means that each geometry comprises one or more geometries, with each internal branch including all polygons of its terminal branches.  The internal boundaries between polygons are not dissolved, so you will see all the component polygons, but any GIS will support a dissolve operation.  If you are wondering what multipolygons and multipoints are, then Wikipedia has a decent explanation as part of the Well Known Text entry.

3.  Remember that Biodiverse knows nothing about coordinate systems and map projections (agnostic might be a better way of putting it).  You will need to define the coordinate system of the output file yourself using a GIS or other geospatial tool.



Shawn Laffan
12-August-2019

If you want to try this out before version 3 is released then the 2.99_005 development release can be accessed through the downloads page at https://github.com/shawnlaffan/biodiverse/wiki/Downloads


--------

For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse


To see what Biodiverse has been used for, see https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList

You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users

Biodiverse now exports tree branch colours

Biodiverse version 2 will allow you to export the tree branch colours.  This means you can now get your tree colours to the point that you are happy with them, and export them to another program such as FigTree to generate figures for a publication.

Colours can be exported with the Nexus and tabular Tree formats.  If you use the tabular format then there will be a column in the output table called Colour.  For the Nexus format, the colours are stored in the comments block for each branch.

Click on the "Export colours" checkbox to export the colours in a Nexus file. 

A few examples are below.

View labels tab with a set of labels highlighted on the tree.

The selection colours displayed in FigTree.

A cluster analysis with six clusters coloured.

The tree in FigTree with the six clusters coloured.  

A user defined selection.

The same user defined selection in FigTree.

The colouring works for anything that is applied to the tree, so you can also export continuous palettes when you have calculations per node (branch).

PD calculated for each cluster node (branch).

And the same PD colouring now in FigTree.

Kudos to Luke Fitzpatrick for getting it working.

Shawn Laffan

26-Apr-2017

For more details about Biodiverse, see http://purl.org/biodiverse 

For the full list of changes in the 1.99 series (leading to version 2) see https://purl.org/biodiverse/wiki/ReleaseNotes (for all issues addressed or being targeted to fix for version 2, see https://github.com/shawnlaffan/biodiverse/milestone/4 ).

To see what else Biodiverse has been used for, see https://purl.org/biodiverse/wiki/PublicationsList

You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users

New selection tool in Cluster analysis tab

A new feature just added to the Biodiverse cluster analysis tab is the ability to control the colour of branches on the tree and the cells that contain them.  This is perhaps most useful if you want to colour your Biodiverse plot to match some pre-existing map (and is the reason some users requested it).

[[ UPDATE 21-Aug-2017.  This feature has been renamed as User defined in the GUI, so wherever you see Multiselect below, you will now see User defined.  ]]

In a nutshell, users can now switch to the Multiselect mode using the combo box where the lists are selected (and the default is still Cluster).  Once there they can choose a colour or accept the system generated default, click on a branch and watch the branch and all of its descendants and the associated groups (cells) plot in that colour.

The multiselect mode is turned on by selecting it in the lower left combo box.  

Users can assign colours to any branch in the tree to colour its descendants and the associated groups. In this example the red clade has also had a sub-clade cleared of colour (note the black branches and the highlighted cells that are not coloured).

Once the branch is selected the default colour changes to the next colour in the palette (unless you turn it off using the button to the left of the brush).  Repeatedly clicking on the branch will cycle through the palette, so if you missed the colour then just keep clicking until it goes around.  The palette in use at the moment has nine colours (it is the 9-colour paired palette from http://colorbrewer2.org).

You can also uncolour branches by selecting the brush icon to change to clear mode.  When in this mode, the mouse icon will change to a brush when a branch is hovered over to remind users what will happen when they click.

There is also little need to fear mis-clicks, as users can undo and redo selections.  Simply press the "u" key on the keyboard to undo one click, and repeat to keep going back.  If you over-do it then you can press "r" to redo and reinstate a branch colour.  Note that the redo list is reset as soon as you colour a branch.


The colour selection uses the same colour selector window as for the shapefile overlays and cell outline colours.

The colour selector can be used to specify your own colours.

Unfortunately the eyedropper selector does not work well on Windows, as it can only select colours from open Biodiverse windows.  This is a limitation of the system.  The workaround is to use a colour selector tool to copy the colour specification to the clipboard and then paste it into the Color name box in the selector window.   A list of possible tools is in this superuser.com question (with the caveat that I have not tested any).

You can also type colour names into the Color name box, and the small sample I tested of the colours at these URLs worked (mostly).  DarkGoldenRod or LemonChiffon anyone?
http://www.w3schools.com/colors/colors_names.asp
https://en.wikipedia.org/wiki/X11_color_names

Shawn Laffan

12-Sep-2016

For more details about Biodiverse, see http://purl.org/biodiverse 

For the full list of changes in the 1.99 series (leading to version 2) see https://purl.org/biodiverse/wiki/ReleaseNotes 

To see what else Biodiverse has been used for, see https://purl.org/biodiverse/wiki/PublicationsList

You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users

More CANAPE - how to restrict your cluster analysis to groups with significant endemism

Two previous posts described more of the background to the CANAPE analysis and how to do it yourself.

[[  2017-08-10 UPDATE - the second part of the condition used an incorrect index name, so the clustering only applied to the palaeo and some of the mixed cells.  This has been corrected in the post.  ]]


What was not covered in those posts was the final step in the Mishler et al. (2014) paper in which the cells classified as palaeo- neo- or mixed-endemism under the CANAPE scheme were clustered to identify groupings with shared branches from the tree.

The aim of this post is to fill that gap.


The way it is done is conceptually simple.  We just need to constrain the cluster analysis to only use cells that are identified as having significant phylogenetic endemism.  The way it is done takes just a few steps, and a bit of typing.


Analyses in Biodiverse can be constrained to work only on a subset of groups (cells) using a definition query (the same as a "where clause" used in database queries).  The way it is currently done is a little more involved than is ideal, largely due to the amount of typing needed, but is not too complex to get working.  (One day we will develop a spatial conditions builder interface to make spatial conditions easier).

So, imagine you have run a spatial analysis called "phylo_end" in which you have told the system to calculate the indices under the  Phylogenetic Endemism and Relative Phylogenetic Endemism, type 2 calculations.  You then ran 999 iterations of a randomisation called CANAPE to assess how often the observed index values were higher than random.  The spatial results object now has an additional list called CANAPE>>SPATIAL_RESULTS which contains the counts of how often the observed value was higher than the randomised values, how many iterations were applied for each index, how many ties there were, and the proportion that were higher than random.  (See the help for more details about the naming scheme).

An example of the setup for the randomisation.

The randomisations have been completed so now the spatial analysis has an output list called CANAPE>>SPATIAL_RESULTS.  This list contains indices that summarise how often the observed values were higher than (or tied with) those calculated using the randomised data.

The next step is to start a new cluster analysis and set our definition query.  The selected basedata has a randomisation output, so you will be warned about possible issues with synchronisation.  This is only a potential problem if you run more iterations of the randomisation, which is not the case here so you can safely ignore the warning.

You will see this warning when you open a new cluster analysis, but can ignore it in this case.  

Set the Metric to be PHYLO_JACCARD (or whatever metric you like - it is your analysis after all) and enter the text below in the Definition Query box (you will need to expand it to enter text).  If your output and randomisation names differ then edit the text as appropriate.

sp_get_spatial_output_list_value (
    output => 'phylo_end', 
    list   => 'CANAPE>>SPATIAL_RESULTS', 
    index  => 'P_PE_WE'
) > 0.95
|| 
sp_get_spatial_output_list_value (
    output => 'phylo_end', 
    list   => 'CANAPE>>SPATIAL_RESULTS', 
    index  => 'P_PHYLO_RPE_NULL2'
) > 0.95



What this does is use the sp_get_spatial_output_list_value() subroutine to access the randomisation indices.  If either of the PE scores for a group using the observed or alternate trees are higher than the randomisations for 95% of iterations (these are the P_PE_WE and P_RPE_NULL2 indices) then that group will pass the test. Only those groups (cells) that pass the test will be considered for the cluster analysis.

Example of the cluster analysis setup, with a definition query specified.  Only those groups (cells) that pass the test will be used in the cluster analysis.
(The text in the image is different from the main text only in whitespace and formatting.  The code is the same).    

Once the analysis is run it will look something like the image below.

Completed cluster analysis.  Cells that failed the definition query were not considered for clustering and are plotted in grey.  

And there it is, a cluster analysis of the CANAPE regions to identify which ones have similar sets of branches from the tree.

Of course, definitions queries are extremely versatile and can be used for all sorts of applications.  You might, for example, want to cluster within a region defined by a shapefile but consider geographic ranges from across a broader study region.

It is also worth noting that spatial analyses can use definition queries, but they affect the analysis differently.  Calculations will only be completed for groups that pass the definition query, but those groups can have neighbours that fail the definition query.  This allows one to more easily analyse data with larger window sizes but with a guard area (data on the edges of the analysis region which, if not considered, could cause biased results).


Shawn Laffan, 19-Apr-2016


For more details about Biodiverse, see http://purl.org/biodiverse

To see what else Biodiverse has been used for, see https://purl.org/biodiverse/wiki/PublicationsList

You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users