Visualizing Usage over Time

Analytics apps don't tell you much about usage behavior. You might be able to see how many users performed an event, or how many times they did it. But none of the analytics packages out there are good at showing you how often people do things. Are they using to-dos once a week? Every day? Only signing into the app once a month but happily paying for years?

Time matters. You can't understand usage without time. So I tried exploring time series as a way to view usage data. The problem is, it's a lot of data. It's easy to count how many users have done something or not. The result is just a number: x%. How do you count how many users have a similar pattern of behavior over time?

The other challenge is intuition. You might get some kind of statistical answer to these questions. But does that deepen your understanding about the behavior? I've wanted a way to see the behaviors directly.

To that end I've been exploring ways to view time series data on user behaviors one by one and also in clusters. The general technique looks like this: 

  • Query logs to get a time series of usage per person or per account. This looks like # events per day, or # days/week with usage, depending on the context.
  • Dimensionally reduce all the time series and cluster them together by similarity, in the spirit of Principle Component Analysis. So far t-SNE is producing the best results.
  • Visually explore the dimensionally reduced map to see individual behaviors and clusters of behaviors.
  • Lastly, once an interesting behavior is identified, work backwards to count how many accounts or users are in the "neighborhood" of the map. Or work backward to the account IDs to recruit for qualitative research like phone interviews.

Early progress looks like this:



Here's a short movie of interactively filtering the dimensionally reduced map:



Tracking Uncertainty of Work

Tasks and tickets aren't enough to track the progress of work. It's crucial to understand not only what's done, but what's unknown. How does the team feel about the unknowns in the work? Are there things that need to be figured out still? Are there things that we might not find a solution for? Or have we investigated all the unknowns already and it's just a matter of execution?

Work is like a hill with two sides. There's an uphill phase of figuring out what to do and how to approach the problem. That’s the climb. After you reach the top, there aren’t anybody ruinous unknowns. You can see down to the other side and finish executing. It's straightforward to estimate and finish the work from that point.



The Hill Chart captures this intuition and reflects it back to the team. Each piece of work that can move independently gets a dot and team members update the position to reflect their current knowledge.


This eliminates risk earlier in the project. Tracking the progress of work from unknown to known sparks deeper conversations about what kind of work needs to be done at what time.

Teams naturally hide uncertainty. The Hill Chart broadcasts each degree of uncertainty as a meaningful step along the way toward finishing the project.

We're currently experimenting with the Hill Chart at Basecamp. For an early write-up, see  the "Uphill Strategies" section of Running in Circles.

Screen Shot 2018-02-14 at 12.34.18 AM.png