Last time we talked about pre-planning, today we’re going to talk about data collection.

Sources of information

Employees

Rule number 1 is always “talk to your people.” When I begin collecting data for a change management process, I always go to my people first. They will have the firsthand insights on how things are going, what’s working, and what’s not working. Depending on how many people I need to talk to, I can set up meetings, send out surveys, or even create a focus group.

In our case, we noticed a recurring theme in weekly meetings. People would discuss what their PODs were up to and how they could keep up with new product development. In some cases, we had slowed down on maintenance work to make room for new product development. It worked for a time and then we had to take a break every few sprints to focus on catching up on maintenance.

Operations

The operations team may have data that could give me insights into the current state or trends. Maybe annual recurring revenue has been declining, maybe there’s an earlier indicator like a decrease in the sales team’s meeting bookings. I can always dig deeper. If there’s recorded sales calls or notes, I may start to discover some reasons there that I can take back to my team.

Maybe there’s HR data that could help. Employee satisfaction, absenteeism, employee turnover, etc.

In our case, the metric to track was maintenance costs. We had been tracking them for a while and had noticed that we had been incrementally reducing maintenance costs with time, but product development needs required more significant cost reduction, in order to reallocate our resources to new product development.

Technology

Consider what has changed in the field of technology. The thing everyone is talking about right now is AI, but even within that there’s a number of different options available. Do I just want a Q&A chatbot, do I want something to write social media posts, do I want it to take in information and summarize it, write code, create a pipeline of data throw a workflow automation tool?

In our case, it was workflow automation tools. Large amounts of our maintenance work could be automated through these tools. AI is very good at classification (sorting items into categories), so we began investigating how we could use those tools to free up our time, create evals for them so that we could closely manage our margin of error, and then push more difficult cases to human reviewers.

Financial

When it comes to financials, I need to be more granular than just tracking revenue. In a company with multiple product PODs, if I ship a feature and revenue goes up, I don’t necessarily know that my new feature is what has grown revenue. Alternatively, if revenue stays flat, I also don’t know if that’s because my feature is no good. Maybe the sales team was at a conference or someone was sick a few weeks ago and there was a gap in the sales pipeline.

Again in our case it was maintenance costs that we were tracking. The costs were improving over time, but new product development required more resources as well. The average cost per data point was also decreasing, but not as fast as our ambition was growing.

Market

After I have considered the other factors, then I can consider the competition. I don’t want to be in a situation where we’re making a change just because that’s how they do it at Google.

In our case, we noticed a lot of companies moving away from highlighting their human curation. A lot of companies were moving towards AI automation of gathering and cleaning data. With the right expertise and care, this can be done, but it must be done carefully or there would just be a continuous stream of garbage data coming in, poisoning any models that would be built on it.

Sources of misinformation

Consider biases

What opinions have I created in the past that I’m still carrying with me? Did I think AI was really bad at something in 2021 and never gone back to see how the field has evolved since then?

In this case, my bias was with AI automation of some of our workflows. We had some success with writing (with human review), but not as much with no-code automations. Another team member brought some of these AI automation tools to our attention and we were able to break those biases and re-evaluate the situation as it really was. Not just the way I thought it was.

Consider the problems underneath the symptoms

Like I said before. Revenue going up or down is not necessarily directly related to the features we have shipped. We always need to dig deeper.

In our case, it was that the maintenance costs and average cost per data point were decreasing, so for a while I thought things were under control. But some people were experiencing stress trying to keep up, some objectives were being scaled back to fit the amount of time we had, and we couldn’t just hire our way out of the problem. So the only solution was to curate more data with the same amount of time and resources.

Wrapping up

Now that we’ve considered our sources of information and misinformation, we know what is possible and what must be done. I need to change the way my function works, so that we can achieve more with roughly the same level of resources.

Next time, we’ll talk about the stakeholders.