Guerrilla tactics for data analytics

4th December 2012

Anybody who's had contact with software development will be familiar with at least some of the methodologies explored over the years, the spiral model, agile development, the waterfall. However useful as these are, applying their principles to Data Analytics is somewhat of a tricky endeavour.

Unlike a traditional software development project there is no compiled code at the end, unit testing is often impractical given the potentially vast volumes of data involved, and each mined insight may require a variety of tools and technologies to reproduce the steps from the first data drop to the final report.

This is coupled with increased scrutiny of data analysis and the expectation that for investigations, data journalism and certain other fields of research, results will be fully reproducible and documented. Managing projects like this requires close collaboration between management and team members to ensure that analyses are tracked, versioned and tested ensuring quality and traceability.

Recently at the Deloitte BIS conference in Orlando, Enda Ridge (Data Insight Services, KPMG Risk Consulting)* and Edward Curry(Digital Enterprise Research Institute at the National University of Ireland, Galway) presented a paper entitled Emerging principles for guerrilla analytics development which details what they believe are the founding principles of building a 'guerrilla' analytics team.

Why Guerrilla Analytics?:

[These types of projects] necessitate a versatile and fast moving analytics team that can achieve quick analytics wins against a large data challenge using lightweight processes and tools.

These teams are assembled and mobilised in a fashion akin to units assembled in guerrilla warfare (albeit without perhaps the implied life threatening risks). This very much describes the types of projects I've been involved in during my career - tight deadlines, client mandated technology and the expectation of a complete audit trail. And that's before we get onto the continuing shifting sea of requirements from the client.

The paper outlines nine principles for the Guerrilla team around the tenets of:

  • Data Manipulation
  • Data Provenance
  • Coding and
  • Testing

These go a good way to addressing the key activities that a project team such as this would be engaged to perform and while the research is in it's infancy it's refreshing to see a new take on these management problems and an attempt to distil some of the authors' learnings into a set of guiding principles.

Principles however, only get you so far and what I would really like is to hear more of other's experiences in managing data projects and whether you identify any of these principles in your own work, perhaps you even have more to share?

*Disclaimer, I currently work for KPMG.

By Niall Napier

Back to the blog

comments powered by Disqus

This Article's Tags

View the whole tag cloud