Firewall Upsell

Background context

Nexus Repository (NXRM) manages binaries and build artifacts across user’s software supply chain. NXRM OSS provides users free access to the artifact repository with universal format support along with basic functionalities. Firewall is another platform offering which users can purchase after converting to a PRO customer. Purchasing Firewall is a helpful addition to NXRM because it stops known and unknown open source risk from being downloaded into user’s repositories.

The Firewall upsell initiative focuses converting our OSS users to PRO customers (paying customers). While going through internal stakeholder discover, we’ve narrowed down to surface Firewall capabilities and features that differentiate NXRM benefits and showcasing metrics that demonstrate Firewall value proposition.

Overview Goal

We want to raise the number of Product Qualified Leads (PQLs) coming from Repository OSS who convert to Firewall customers.

Challenge

How do we do this iteratively with low-cost initial motions to discover whether higher-effort work will be worthwhile? At each step, teams should have a data-based recommendation for whether continued investment is likely to lead to a higher return on the marginal investment (e.g if we spend another 6 dev weeks on the such and such graph, we believe we’ll get another x Firewall conversions per month which is even more effective than the Outreach static page which lead to x Firewall conversions per month).

Role

As a UX Designer, I provide support to the UX Manager facilitating the Miro board activities. In addition to ideate, help create minimal experiments, and test the hypothesis to determine which best avenues to spend more time on.

Duration

6 months

Technologies used

Miro, Sketch, Google Optimize

We begin with the hypothesis which was based on previously initiatives and internally based research:

• Knowing latest vulnerabilities will make users want to upgrade to Firewall.

• Learning more about Firewall benefits will make users want to purchase Firewall.

• Knowing how likely their repositories can be exposed to risks will prompt the users to upgrade to Firewall.

• Knowing current risks in their existing repositories will make the users upgrade to Firewall.

• Knowing their current Log4J risks will prompt more Firewall uptake.

Previously proposed solution

Past research and designed solutions from other initiatives provided insights on the current proposed solutions for the Firewall Upsell initiative. Facilitated group meetings also provided us an idea of what the effort estimation would look like. We match up the previously proposed solutions and ideas with the hypothesis to ensure we’re on track to validate our assumptions.

Plot Solutions: determine the least amount of developer weeks to raise PQLs

Additionally, plotting the solutions will allow us to see the amount of PQL versus the dev weeks it takes to test each experiment. This will also shine a light on the least amount of dev weeks it will take in order to see an increase of product qualified leads.

We gathered that the least amount of dev weeks and some uptick of PQL showed 4 solutions: 

1. Static CTA explaining Firewall and benefits

2. Informational. Profile newest vulnerabilities

3. Point users to Log4j visualizer and tie messaging into what we already have

4. Automatically show metrics on risk within your existing repositories

Collaborating with project managers, stakeholders and the marketing team provided their suggestions on these experiment ideas corresponding to the previously proposed solutions.

We concluded with experimenting on 5 different variants to determine which CTA and its content that would encourage users to learn more about Firewall by utilizing a tool our company is familiar with: Google Optimize. We believe using the multi-variant testing within Google Optimize will determine a series of minimal experiments we can best test and track in the product so we know which avenues would be best spent more time on.

With Marketing’s help we narrows down 5 different messaging to test with - 4 out of 5 messaging, we will be testing due to the fact that the fifth messaging requires additional investigation from the development side.

Completed Banners for A/B testing:

Designing the 5th variants

Through another follow up conversation, we have discovered another use case of which if neither of these formats are in use within their instances - we will show the user’s a different visual with the same copy and CTA as the 5th variant.

Next steps

The variants and goals have been set up through Google Optimize for testing. Next steps would be to synthesize the data we gather in one months time. This will also provide us additional time for the developer’s team to investigate on the 5th variant on how to surface user’s data to the Outreach page. We will then determine which variant tested well and test the successful variant with the 5th variant. As this is an iterative process, the data will inform us on whether continued investment is likely to lead to a higher return on the marginal investment.

Update

A full 30 days time period has passed and we were able to synthesize the data we collected from Google Analytics and Google Optimize. Meanwhile, the investigation has been completed and we are now able to surface the first 3 most popular downloaded format (npm, PyPi and Maven) within their repositories to the Outreach page.

Challenges and frustrations

Goals have been created to track the conversion rate when users are clicking the CTA button. UTM links are in placed for each variants’ button (total of 4 for now) to distinguish the data for each and to determine which variant has been more successful in gathering clicks. Unfortunately the conversion rates for our goals set up are not returning within Google Analytics. We are manually converting the rates based off users’ visits (ad views) and clicks and we are also getting PQL rates from the Marketing team.

Overall Experiment Results

Conclusion

Based on the data, Outreach marketing hasn’t generated a high amount of traffic to the Firewall page. Even after users had visited the Firewall page, the amount of PQLs generated is 14 which is extremely low. Due to insufficient evidence, we concluded to spend the development effort allocated for this initiative to other priorities.

Even though this initiative was put to a stop, we were able to move forward with a data-driven decision and the results showed us if these investments are likely to lead to a higher return on the marginal investments. From this, we helped our C-Suites reprioritize and reallocate development weeks to higher priorities.

< Back