Data Mining and the Analysis of Large and Complex Datasets

Almost all public and private sector organisations routinely collect a wealth of data concerning sales or service provision. A significant proportion (particularly in the public sector) fail to recognize (or at least fail to exploit) the potential importance of the information embedded within that data - information that could, for instance, be used to improve or better target the services or products provided.

Extracting and analysing activity-relevant information from what are often large and complex datasets is seldom straightforward, particularly when value is added primarily through integration with other internal and/or external datasets. For instance, it is estimated that over 80% of the routine service data held by public sector organisations is 'postcoded'. This means that the 'transactions' described are given fairly precise locations - this commonly being the home address of the recipient of the service being provided. Constraints of confidentiality and privacy means that postcode data must be treated with particular care but, suitably anonymised and aggregated, they can reveal important insights into the the geography of service provision. Even more can be achieved when such data are linked to other spatially-referenced data, such as those provided by the 2001 Census. In such a way it would, for instance, be possible to assess the extent to which socio-economic or other characteristics affect the demand for services. Taken a step further, the analysis of outliers can reveal where service uptake appears relatively high or low, thereby raising issues about service access and equity. Many public sector organisations now have a statutory duty to formally address these issues.

More complex and nuanced mechanisms by which to assess the extent to which public sector organisations are meeting the needs of local communities are considered elsewhere, but much can be achieved simply through a detailed analysis of routinely collected data. 'Data mining' is the term given to this process, and RAE Consulting can advise on the identification and extraction of activity-relevant information from your routine data, and on what opportunities exist to add value to your own data through integration with other datasets. We have particular experience and expertise in postcode-census linkage algorithms.

Please contact us if you would like to discuss further how best to extract and analyse the data you already collect.