It all started with a curry and a beer.
Nigel and I were midway through a 12 month Business Intelligence project was integrating over 40 data feeds sourced from CSV and spreadsheets. Between human error and changes by external providers that the hub would still fail regularly and there was nothing that could be done other than patch the error and re-run the load. So, we nipped out for lunch at an Indian restaurant to clear our heads. Over the course of talking about the project, we kept coming back to the same question: “what can we do to stop businesses from stumbling because of inconsistent data?”
Over our combined 25 years of experience, every client, no matter the size, still has their business processes dependent on data they can’t fully control. This causes delays, increased costs, and despite the increased availability of data, we didn’t see IT methodologies working to solve this problem. Quite the opposite, data management assumes that changed will be managed, even though we all know this is often not the case.
So, we tucked into our mains, ordered another beer and talked about how we would fix this in an ideal world. One where you don’t assume the data will not change format (in fact, you assume it will), and where business users who know their data can manage it, not external vendors or IT staff working to time pressures. Realising we either had come up with a really good idea that nobody else had tackled, or a really bad idea that you’d be crazy to do, we walked back to work in the rain and started planning.
Conductor is the outcome of that meal, where we wanted a system that made transferring data between IT systems so easy anybody can do it. And so fast that you can push spreadsheets into databases before your coffee gets cold. We assume that nothing is fixed, so it had to be smart enough to pick up and fix the common errors (typos, changes in data formats, copy-paste errors) without impacting on downstream systems. And finally, we wanted our customers to have the data management tools they deserve: simple, pragmatic, and easy to use.
