Holiday tinkering: ChatGPT to analyze Streaming promotions

This break, I was faced with a challenge: I run a market data company so, naturally, I have access to tons of exciting data that could potentially move markets, drive decision-making, and illuminate leaders at subscription businesses like Netflix and Peloton.

However, one of the most common problems is the more data = more noise phenomenon. And, like many of our customers, I was faced with that exact problem last week: we had a fresh batch of really interesting data but I didn’t have access to a smart analyst who could pull out key insights.

My use case: I wanted to create a simple chart with one compelling insight from a sea of data and email that to prospects to showcase a new product launch.

For any data company, product marketing is done much more effectively when you are using the data to highlight key insights (indirectly proving the value of your product) rather than trying to describe the product’s key features & benefits (as is done in traditionally enterprise product marketing). You need to show, not tell.

So, let’s dig in…

The data & analysis

Note: much of this is proprietary data, so I haven’t shared the original file nor the ChatGPT prompt publicly.

The new data I had access to was a list of all major promotions run by a streaming service: key metadata, number of Sign-ups, Retention rate, and so on. This data was on several discrete tabs, so it was very difficult for me to create an easy pivot table. I started out by asking ChatGPT to help me manipulate all of this in Excel, which it helped with…

…but then quickly realized — wait a minute, why not just treat ChatGPT less like CoPilot and more like the analyst itself? Once I had that realization, things got going. ChatGPT was able to ingest the data well and do a couple important things:

Ingest data and describe schema across 4 separate tabs; and understand how each tab related to each other.

Manipulate columns to make it possible to connect these tabs together for deeper analysis. This was an extremely important step because it had to disaggregate the 4 separate tabs using a synthetically created primary key such that I could work with one single table.

Run analyses on many of my key questions such as:

• Which promotion had the most Sign-ups? On day 1? In total?

• What about promotions that were specifically run via the Amazon Prime Video Channels ecosystem?

• Did promotions of ad-free tiers do better or worse than those of ad-supported tiers? How about monthly vs. annual?

• Show me the Retention Rate for those top promotions. Can you put it in a simple scatter plot?

What I learned: Just like any top tier analyst, ChatGPT is excellent at quickly manipulating disparate data into a ready-for-analysis format, and then acts instantly as a conversational analyst – translating & abstracting your prompts into complex SQL queries and/or Excel formulas – and then returning the data to you. And it doesn’t take PTO, have trade-offs on what it can work on, and other “human” problems.

Accomplishing my goal

I decided that the last chart (above) would make a juicy insight for a prospect, so I asked ChatGPT to craft a quick email. I don’t have any integrations set up, so it wasn’t able to interact with the “outside world” — and so this is as far as I got. But I was able to copy / paste this into an email and hit send.

So, mostly, mission accomplished! I was able to do effective product marketing for our new data product by highlighting a key insight in the sea of data, rather than just a list of cool features. And I didn’t need any technical skills nor teammates to help me.

Next steps: going further

As I was going through this, there were a number of ideas that popped up that I didn’t have the data access, time, or wherewithal to accomplish (and some of it is beyond ChatGPT’s current functionality).

• Build query templates: there are a series of obvious questions any client-facing employee would ask of their client at our company. How many Sign-ups did they generate? How high was Churn? How did it change from last month? What about vs. their peer set? We should have a company-wide instance set up on top of basic data with dozens of query templates to get this information readily with one click.

• Integrate into sales workflows such that I could easily put this into gmail, Apollo.io drip campaigns, and so on. This type of simple prospecting could be easily automated from end-to-end.

• Upload Antenna corporate branding such that the charts were instantly placed into our style guide.

• Build case studies: this could be a major boon for any data company (or, company that sits on top of a lot of proprietary data). Benchmarking reports, client case studies, and other similar sales collateral could be easily produced even when the product marketing team doesn’t have ready access to analytics resources.

• Feed even more data: most of the shortcomings from this experiment (outside of user error) were driven by two things: (1) lack of data availability and (2) data was too heavily aggregated. If both of those conditions were reversed, the amount of insight I’d be able to glean would be an order of magnitude higher than what I achieved. For example, I’d be able to use the following logic path with endless permutations:

  • Show me how [KPI of choice] changes over time or vs. peer group

  • Explain to me how [KPI of choice] is explained by [variables of choice]

    • Of course, this is basic Data Science work – the point is not that I’m creating new statistical methods, the point is that I’m accomplishing them without the technical and human resources I’d be otherwise dependent on.

Bigger picture: rewriting analytics workflows

As I was doing this, I had a realization: wow, this is going to completely rewrite existing analytics workflows, which currently look something like this:

How will this rewrite the above workflow?

• Eliminate the role of a standalone analytics team in favor of more embedded subject matter experts at the business user groups. If you know your query, and you don’t need the same SQL skills to get the answer, you’re a whole lot less dependent on your Analytics team. The “democratization of data” trend is only going to continue.

• Turn data engineers into prompt engineers with a specific focus on doing ETL tailor-made to create schemas that ChatGPT will easily understand. My little experiment works really well for clean, structured data — but it’s not clear how much structuring would be required for messier data. Historically, creating these schemas has been the key role of data engineering. That doesn’t change but in the future, it will be done specifically with the intent of creating a query-able LLM.

• Much more personalization and parallel workflows: without the human resource constraints, why wouldn’t every case study or benchmarking report be personalized? As long as the data exists, this technology makes it possible.

• Workflow tools will be created to completely abstract everything between “properly structured data” and “intelligent query” with a high degree of precision.

The bottom line: a couple high-level takeaways

1: With ChatGPT, we see a familiar cycle repeating itself: all technical skills are rapidly commoditized; therefore, a technical specialist has only three alternatives: (1) continue to stay at the cutting edge of new technologies, (2) develop domain expertise in a certain industry, or (3) get left behind. For what it’s worth, this is my #1 piece of advice to all technical specialists. Think about web development. What was once highly skilled coding has now been abstracted by Squarespace and Wix.

2: Those who have strong first-party data just got the world’s biggest turbo boost. As long as you can structure it to be consumed by an LLM, you can pull insight from it. Think about a consumer subscription company such as Duolingo or Peloton. A CEO, a CFO, a Head of IR, a Head of Marketing… all of these folks can now simply ask the data questions like:

• Can you show me the list of our 99th percentile most retained users? What usage data factors most separate those users from the rest? Can you craft that into a paragraph I can use on today’s earnings call?

• For users who came in through a specific ad campaign, can you show me which SKU they signed up for? And for the ones who signed up to the most expensive SKU, can you trace which instructors they used in an easy to understand pie chart?

  • None of these analyses, in themselves, are particularly unique. What’s unique is the sheer volume of requests that can be pushed through without the previous constraints. When it’s cheap & easy to get an insight, you don’t think twice.