Thursday, 12 November 2020

Randomisations now also generate z-scores

Randomisations in Biodiverse have always generated rank-relative scores from which significance can be derived. See for example this previous post: https://biodiverse-analysis-software.blogspot.com/2016/08/easier-to-use-randomisation-results.html 

However, sometimes users want to use z-scores to assess a randomisation. Several other tools such as phylocom also calculate z-scores, and often these are the default.

From version 4 of Biodiverse, z-scores are now also be calculated for randomisations. More details about what happens are in the updated help page, but a short summary is given here. 

For each iteration of the randomisation, each analysis is regenerated using the same set of spatial conditions, but using the randomised version of the basedata. The values of each index for each group (cell) are then checked, and running tallies are kept for the sum of randomised index values as well as the sum of squared index values. These sums, along with the count of iterations, allows the calculation of the mean (sum of x / count) and standard deviation using the running variance method. 

Once the randomisations have completed, the z-score for each index value for each group can be calculated in the usual way as (observed - expected / sqrt (variance)). Interpretation is the same as any other z-score, with values outside the interval [-1.96,1.96] being significant for two tailed test with an alpha of 0.05, providing the number of samples is large. Note that the strictly correct value depends on the degrees of freedom, which depends on the number of samples. But why would you run a small number of iterations? 

The z-scores are calculated automatically whenever a randomisation is run.  The results are collated for each group and are accessible through the ">>z_scores>>" lists.  In this case Rand1 is the name of the randomisation, hence the list name is Rand1>>z_scores>>SPATIAL_RESULTS. 



If you want to add z-scores to an existing randomisation then you will need to run at least one extra randomisation iteration to trigger their calculation. If there is demand to do so without the extra iteration then we can add an option for it.  (Consider the relative numeric difference between 999 and 1000 iterations, without worrying about aesthetics such as the number of observed plus random realisations being a multiple of 10).  


Shawn Laffan, 12-Nov-2020



For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse/ 

For the full list of changes in the 3.99 development series, leading to version 4, see https://github.com/shawnlaffan/biodiverse/wiki/ReleaseNotes#version-399-dev-series 

To see what else Biodiverse has been used for, see https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList 

You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users

Friday, 6 November 2020

Updated handling of Cluster and Region Grower analyses in randomisations

Randomisations in Biodiverse are used to assess the statistical significance of a set of analysis results given some randomisation scheme such as shuffling the species around the map (labels across groups in Biodiverse terms), subject to constraints such as each cell (group) must maintain the same number of species. Randomisations are key to interpreting where results differ from what would be expected, and are integral to protocols such as the Categorical Analysis of Paleo and Neo Endemism (CANAPE). 


The basic process of the randomisation is this.  For each iteration of the randomisation analysis, Biodiverse will:
  1. Create a new basedata object with a random allocation of labels to groups
  2. For each analysis in the basedata 
    1. Regenerate a version using the randomised basedata
    2. Compare the values of the analyses from the original and randomised basedata and track if they are higher or lower, on a cell by cell basis
    3. Track basic statistics of the distribution to allow the calculation of the mean and standard deviation, and thus z-scores (this is new in Version 4 - there will be more on this in another post).

The tracking is used to reduce memory usage, as the randomised basedatas and outputs can be discarded as soon as they have been collated.  This is more efficient than keeping all the results in memory for later ranking, especially with large data sets such as those used for Mishler et al. (2020).

A huge amount of work has been spent over the last several years to make the randomisation process go faster and scale better with larger data sets, with recent versions being orders of magnitude faster than some of the earlier versions. In some cases analyses now take minutes where they used to take days. However, one slow-down that still affects things in version 3.1 is the effect of tree structures in a basedata. These are the Cluster and Region Grower analyses. 

The reason the Cluster and Region Grower analyses slow things down is simply that they take longer to calculate.  A spatial analysis only need to pass over each processing group (cell) once, so one can think of it as N calculations.  In comparison, a cluster analysis will compare each group with each other group to create its matrix.  If one is clustering a full data set then this will be N(N-1)/2 calculations, but the same scaling effect applies for subsets defined using definition queries (e.g. for CANAPE).  Even when one considers that the spatial analysis might include many neighbours for each group, the Cluster and Region Grower analyses take much longer.  And then the Cluster analysis needs to process the matrix.

Even with the slow-down, a comparison of the randomised Cluster or Region Grower analysis with the original is not always informative.  

In Biodiverse Version 4, this process has been changed.  Cluster and Region grower analyses are now skipped by default.  This will substantially speed up randomisations containing such analyses.  

If you still want to run them then there is an option to do so.  See the red arrow in the screenshot below.  


Randomisations of Cluster and Regions Grower analyses is off by default in Version 4, but can be re-enabled if needed (see red arrow).



One point to note is that any calculations per node (branch) are still done.  This was modified some time ago to use the set of randomised groups under each branch from the original, non-randomised tree.  This works because the group IDs are the same across both the original and the randomised basedatas.  It is just that the randomised version has a randomly allocated set of labels. 



Shawn Laffan
06-Nov-2020



For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse/ 

For the full list of changes in the 3.99 development series, leading to version 4, see https://github.com/shawnlaffan/biodiverse/wiki/ReleaseNotes#version-399-dev-series 


To see what else Biodiverse has been used for, see https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList 


You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users 

Tuesday, 24 March 2020

Publications using Biodiverse in 2019

2020 is now in full swing, so here is a list of publications from 2019 that used Biodiverse. 

If you want to see the full list (111 at the time of writing), then go to https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList 

For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse/ 

Shawn Laffan
24-Mar-2020



Allen, J.M., Germain-Aubrey, C.C., Barve, N., Neubig, K., Majure, L.C., Laffan, S.W., Mishler, B.D., Owens, H.L., Smith, S.A., Whitten, W.M., Abbott, J.R., Soltis, D.E., Guralnick, R. and Soltis, P.S. (2019) Spatial phylogenetics of Florida vascular plants: The effects of calibration and uncertainty on diversity estimates. iScience, 11, 57-70.

Crouch, N.M.A., Capurucho, J.M.G., Hackett, S.J. and Bates, J.M. (2019) Evaluating the contribution of dispersal to community structure in Neotropical passerine birds. Ecography, 42, 390-399.
Dagallier, L.-P.M.J., Janssens, S.B., Dauby, G., Blach-Overgaard, A., Mackinder, B.A., Droissart, V., Svenning, J.-C., Sosef, M.S.M., Stévart, T., Harris, D.J., Sonké, B., Wieringa, J.J., Hardy, O.J. and Couvreur, T.L.P. (2019) Cradles and museums of generic plant diversity across tropical Africa. New Phytologist.

Daru, B.H., le Roux, P.C., Gopalraj, J., Park, D.S., Holt, B.G. and Greve, M. (2019) Spatial overlaps between the global protected areas network and terrestrial hotspots of evolutionary diversity. Global Ecology and Biogeography, 28, 757-766.

Fuentes-Castillo, T., Scherson, R.A., Marquet, P.A., Fajardo, P., Corcoran, D., Román, M.J. and Pliscoff, P. (2019) Modelling the current and future biodiversity distribution in the Chilean Mediterranean hotspot. The role of protected areas network in a warmer future. Diversity and Distributions, 25, 1897-1909.

Gallien, L., Thornhill, A.H., Zurell, D., Miller, J.T., and Richardson, D.M. (2019) Global predictors of alien plant establishment success: combining niche and trait proxies. Proceedings of the Royal Society B, 286, 20182477.

Garcia–R, J.C., Gonzalez-Orozco, C.E. and Trewick, S.A. (2019) Contrasting patterns of diversification in a bird family (Aves: Gruiformes: Rallidae) are revealed by analysis of geospatial distribution of species and phylogenetic diversity. Ecography, 42, 500-510.

Greenlon, A., Chang, P.L., Damtew, Z.M., et al. (2019) Global-level population genomics reveals differential effects of geography and phylogeny on horizontal gene transfer in soil bacteria. Proceedings of the National Academy of Sciences, 201900056.

Lam-Gordillo, O. and Ardisson, P-L. (2019) The global distribution and richness of frog crabs (Raninoidea). Crustaceana, 92, 1-31.

López-Aguirre, C., Hand, S.J., Laffan, S.W., & Archer, M. (2019) Zoogeographical regions and geospatial patterns of phylogenetic diversity and endemism of New World bats. Ecography, 42, 1188-1199.

McDonald-Spicer, C., Knerr, N.J., Encinas-Viso, F. and Schmidt-Lebuhn, A.N. (2019) Big data for a large clade: Bioregionalization and ancestral range estimation in the daisy family (Asteraceae). Journal of Biogeography, 46, 255-267.

Mekala, S., Donoghue, M.J., Farjon, A., Filer, D., Mathews, S., Jetz, W. and Leslie, A.B. (2019) Accumulation over evolutionary time as a major cause of biodiversity hotspots in conifers. Proceedings of the Royal Society B, 286, 20191887.

Shelley, J.J., Dempster, T., Le Feuvre, M.C., Unmack, P.J., Laffan, S.W. & Swearer, S.E. (2019) A revision of the bioregionalisation of freshwater fish communities in the Australian Monsoonal Tropics. Ecology and Evolution, 9, 4568-4588.

Tolley, K.A., da Silva, J.M. & Jansen van Vuuren, B. (2019) South African National Biodiversity Assessment 2018 Technical Report Volume 7: Genetic Diversity. South African National Biodiversity Institute, Pretoria. http://hdl.handle.net/20.500.12143/6376

Friday, 13 March 2020

Biodiverse version 3.1 has now been released

Biodiverse version 3.1 has now been released



Versions for Windows, Mac and Linux are available and can be downloaded from https://github.com/shawnlaffan/biodiverse/wiki/Downloads

Installation instructions are at https://github.com/shawnlaffan/biodiverse/wiki/Installation


Version 3.1 represents 7 issues closed across since version 3.00 was released.

Highlights of the changes since version 3.00 are at https://github.com/shawnlaffan/biodiverse/wiki/ReleaseNotes#version-31

A more detailed listing of the closed issues is at https://github.com/shawnlaffan/biodiverse/milestone/16?closed=1


The main changes of note are that the View Labels lists now sort properly again (see issues #744 and #747), and that the Select and Pan buttons work again on Mac binary releases.



Shawn Laffan
13-March-2020


--------

For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse


To see what Biodiverse has been used for, see https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList



You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users

Tuesday, 3 September 2019

Biodiverse version 3.00 has now been released

Biodiverse version 3.00 has now been released


Versions for Windows, Mac and Linux are available and can be downloaded from https://github.com/shawnlaffan/biodiverse/wiki/Downloads

Installation instructions are at https://github.com/shawnlaffan/biodiverse/wiki/Installation


Version 3.00 represents 44 issues closed across 350 source code commits.  187 files have been changed, with 13,623 insertions and 10,486 deletions.

Highlights of the changes since version 2.1 are at https://github.com/shawnlaffan/biodiverse/wiki/ReleaseNotes#version-300, and the related blog posts can be accessed via https://biodiverse-analysis-software.blogspot.com/search/label/Version3

A more detailed listing of the closed issues is at https://github.com/shawnlaffan/biodiverse/milestone/15?closed=1


The only change of note since the most recent development release (version 2.99_005) is to address issue #742.  The internal index used for matrices now uses the C locale for numeric values. Incorrect values could otherwise be returned in some locales where the comma is used as the radix character. Biodiverse now throws an exception when it encounters indexes with commas in the values, recommending that the matrix be rebuilt.



Shawn Laffan
03-September-2019


--------

For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse


To see what Biodiverse has been used for, see https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList



You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users

Monday, 12 August 2019

Version 2.99_005 released, Version 3 is next

The final development release in the 2.99 series has now been released (2.99_005).

Barring any major issues, this will be the final development release before Version 3 is released.


The main update in this version (from 2.99_004) is that the spatial patterns of cluster and region grower analysis outputs can now be exported to shapefile format, with each branch getting its own polygon or point features.  This means you can now use the polygon or point geometries for any branch in the tree for further analysis and/or display.  More details are in the blog post:  https://biodiverse-analysis-software.blogspot.com/2019/08/export-cluster-analyses-to-shapefiles.html


The more detailed summary of changes in the 2.99 series is at https://github.com/shawnlaffan/biodiverse/wiki/ReleaseNotes#version-299, and the related blog posts can be accessed via https://biodiverse-analysis-software.blogspot.com/search/label/Version3


This version can be downloaded from links on the downloads page at https://github.com/shawnlaffan/biodiverse/wiki/Downloads


Shawn Laffan
12-August-2019


--------

For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse


To see what Biodiverse has been used for, see https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList



You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users

Export cluster analyses to shapefiles

In Version 3 of Biodiverse it is possible to export your cluster and region grower analyses to shapefile formats.

This makes it easier to use the regions in geospatial analyses, for example as conditions in spatial analyses in Biodiverse.  Perhaps the best example is that you can generate a regionalisation using one data set, and then assess its values for one or more other data sets.  See for example González-Orozco et al. (2013, 2014a and 2014b).

The option is available via the export menu for cluster and region grower analyses.  The options are all the usual ones, although that is not many in this case.


The option is chosen from the export menu for a cluster (or region grower) analysis.  


There are not that many choices, but Shape type can be exported as POLYGON or POINT geometries (shapes).  You can attach lists of results, or choose to only export the geometries themselves.


You might need to run a selection to highlight the branch you are interested in.  This example is branch 120___, which corresponds to the red cluster in the first screenshot above.  You might also choose to use a Definition Query (ArcGIS) or Layer Subset (QGIS) to filter out the geometries you are not interested in.  


And here the component polygons that form the polygons for branch 120___ are highlighted in red.



There are three points to watch for.

1.  If you export a list with the data, then each polygon or point feature is repeated for each item in the list.  This is due to shapefile field names being limited to 11 characters, which is far too short for many of the list entries in Biodiverse.  (Support for the GeoPackage format is in the works to obviate this issue).

2.  The exported features (geometries) are multipolygon or multipoint.  This means that each geometry comprises one or more geometries, with each internal branch including all polygons of its terminal branches.  The internal boundaries between polygons are not dissolved, so you will see all the component polygons, but any GIS will support a dissolve operation.  If you are wondering what multipolygons and multipoints are, then Wikipedia has a decent explanation as part of the Well Known Text entry.

3.  Remember that Biodiverse knows nothing about coordinate systems and map projections (agnostic might be a better way of putting it).  You will need to define the coordinate system of the output file yourself using a GIS or other geospatial tool.



Shawn Laffan
12-August-2019

If you want to try this out before version 3 is released then the 2.99_005 development release can be accessed through the downloads page at https://github.com/shawnlaffan/biodiverse/wiki/Downloads


--------

For more details about Biodiverse, see http://shawnlaffan.github.io/biodiverse


To see what Biodiverse has been used for, see https://github.com/shawnlaffan/biodiverse/wiki/PublicationsList



You can also join the Biodiverse-users mailing list at http://groups.google.com/group/Biodiverse-users