About the product

In 2023, I worked on improving Hotmart's Checkout, the company's first and most established product, focused on selling digital products. The Checkout was already recognized for its robust and advanced technology, particularly its loading speed, which was crucial for both user experience and sales conversion. Given the critical impact of performance on the product's success, there were limited opportunities to make changes to the user interface without compromising this efficiency.

Additionally, the Checkout offered extensive customization options for clients, allowing each to configure their own visual identity and elements in the payment form. While this flexibility was beneficial, it also presented challenges, as the wide range of configurations could impact consistency and usability.

Since the Checkout served clients globally, staying attentive to the implementation and performance of new payment methods was crucial. It was essential to ensure that the existing payment experiences were functioning smoothly and continued to perform optimally across different regions.

Impact

More than 80% of users, on average, who entered the checkout did not complete their purchases. This percentage was even higher when it came to users from other countries.

My role

As a Product Manager, I faced the challenge of using much of what I had learned as a Product Designer and applying my passion for data, which influenced many of my decisions at that time.

Context/Problem

Goal

Increase checkout conversion by implementing low to medium-effort improvements with a quick impact on the payment flow, without affecting performance, and applicable to all types of checkout.

Team

Product Manager (me), Product Designer (1), Front-end (2), Back-end (2), Data Analyst (1)

Process

As a Product Manager, I participated in almost every stage, from discovery to analyzing and monitoring results. However, I was more involved in some activities than others, so I categorized my involvement in each stage of the process as Directly involved, Following closely (I was not actively participating), and Supported (I provided assistance in some activities).

Prioritization and Definition

We worked with a backlog that was fed by various sources, including user surveys, analysis of user behavior data, studies of frequent issues reported to the support team, market research, and more.

At that time, however, we needed to focus on low-effort, quick-return actions. Therefore, we prioritized tasks using an Impact vs. Effort Matrix, which helped us identify improvement opportunities based on quantitative data and the context provided by research.

Often, research was also prioritized when there was an opportunity, even if it wasn't ready for development. This way, we gained more clarity on the scope of future work.

Definitions

This way, we identified three hypotheses with the greatest potential to increase conversion, which would be tested, since every new experience implemented in the checkout should have been previously validated through an A/B test.

• Problem #1 Smart Payment Method: In Brazil, our largest market, the first hypothesis was to offer the most commonly used payment method within a price range, based on data from the past six months.

• Problem #2 Improved payment flow: In Colombia, the second largest market, the goal was to improve the payment flow of a recently implemented method that was gaining traction but seemed to have some issues with payment approval recently.

• Problem #3 Fast Buy: Additionally, in Brazil, the idea was to streamline the repurchase process by offering a quick checkout option using tokenized cards and previously saved address information, removing friction during the purchase. Since we did not require a login to complete purchases, many users were unaware they could use data from previous transactions, even though we had a high percentage of returning buyers in the Checkout.

Discovery

For the second item, Improved Payment Flow, we identified that the low approval rate for the NEQUI payment method occurred mostly on Mobile devices, which accounted for almost 90% of all access. We also noticed that the drop intensified in the last week. Our Product Designer found several comments in the app stores suggesting that the issue was likely caused by a recent update to the app, which affected the payment completion process. However, the app's overall rating remained positive.

For the first prioritized item, Smart Payment Method, we did not conduct additional discovery beyond the initial analysis. In that analysis, based on data reported by the technical lead, we identified the most frequently used payment methods by customers over the past six months, segmented by device and product price range.

Additionally, we performed some benchmarking on major e-commerce platforms in Colombia to observe how they communicated and structured their payment flows. We also watched several videos explaining how this particular payment method worked, as it was unfamiliar to us as Brazilians.

For the last prioritized item, we had been following market trends, especially internationally, around one-click buy solutions. According to market data, one-click buy offered many advantages, such as reducing cart abandonment (due to faster payment), decreasing form input errors, shortening the purchase time, and increasing user satisfaction during the checkout process.

We analyzed repeat purchase data from the checkout system over the last year and found that 70% of buyers had made repeat purchases. This meant that 70% of users had to fill out the entire form again, including their credit card details. Most of these purchases were made with credit cards. We used this data as a starting point to dive deeper into how we could test this solution.

Even with the hypothesis of the app's update being faulty, we reevaluated the payment flow journey for this method to identify areas for improvement, especially since the approval rate on mobile had been low even before the app update. To help with testing, we involved a few Colombian customers who reported difficulties completing their purchases, reflecting the issues found in the app store reviews.

We kicked off with a brainstorm session led by me, not only with the team but also with managers and directors, to align our understanding of one-click buy, its benefits, and different types of flows. This helped clarify the paths we could take, including security steps and the importance of brand recognition in this process.

I also presented a benchmark analysis of several well-known market tools that had already implemented one-click buy functionality, highlighting the specific characteristics of each. We discussed strengths and weaknesses, documenting the user experiences we could leverage for our tests.

During this phase, I worked closely with the Product Designer on the team, getting directly involved in both benchmarking and brainstorm sessions. We tackled all three prioritized items simultaneously, distributing tasks primarily during the discovery phase, whether they were technical or design-related.

Problem #2 Improved payment flow

Problem #1 Smart Payment Method

Problem #3 Fast Buy

Ideation

Problem #1 Smart Payment Method

Problem #2 Improved payment flow

In the first initiative, there was essentially no need for creation. The only change we would make was to present the PIX payment method for specific products in Brazil, within a certain price range. This would eliminate one of the main reasons for cart abandonment at checkout, which was the choice of payment method. The changes happened on Mobile and Desktop.

In the second initiative, the focus was on Mobile. Besides making subtle interface changes, we focused on improving the copy that explained the importance of providing accurate information and clearly outlined the steps users needed to follow to complete their payment. We also worked on improving the copy for cart abandonment emails.

For the third item, Quick Purchase, we created three versions for testing. In these, we explored options where users could authenticate using Google Account (1), log in with a Hotmart account (2) (which was previously difficult to create directly from the checkout), and a "seamless" login triggered by entering the email (3). All methods used two-factor authentication for security purposes.

Problem #3 Fast Buy

I was involved only in the concept development of the following interfaces, they were designed by the Product Designer on the team.

Validation

Even though it was an A/B test, we learned through other initiatives that collecting feedback beforehand, in a quick and cost-effective way, helped us eliminate small details that we would otherwise only notice when analyzing the test results. The process always involved sending the changes through a Design Review, which was already a part of our structure. As the Product Manager, I participated in this review, after which I would send a guided test prototype to users recruited via a survey we conducted within the checkout, just before the test.

This test was built using Maze, a tool that provides valuable data on user actions during testing, and also allows us to combine tasks and questions.

A/B Testing - Delivery

Based on the feedback from both the Design Review and the Maze tests, the development team would receive a refined prototype for test implementation. For the testing itself, we used the company's in-house tool, which distributed visitors according to the percentages and regions we defined and presented the winning audience based on conversion rates.

In addition to this standard process, I introduced the step of including user interaction events at different stages of the test. We found it difficult to draw conclusions from conversion numbers alone, so we needed to understand if users were facing additional friction with the new experience.

To achieve this, I created a flow diagram of the test, pinpointing key moments where it was important to track what the user did.

To track the results, the company's Data Science team worked on integrating the data and calculations, making it easier for us to view the test results in our internal tool. What was initially a time-consuming process became much more accessible for anyone who wanted to follow the outcomes. We were able to monitor session data, conversions, uplift, and confidence intervals.

With everything integrated into our database, we were able to conduct in-depth analyses of user behavior in the tested experiences.

Results and Learnings

• We successfully achieved 2 out of 3 of the prioritized initiatives, resulting in a 0.9% increase in conversion rates. While this number may seem small, it represented a significant impact on our overall revenue, particularly in a highly competitive market.

• Managing tasks that directly affect GMV (Gross Merchandise Value) can be stressful, which is why clear alignment across the team is critical. We made sure everyone understood each step, the overall goals, and consistently provided data to support and argue our case to leadership. It's important to remember that an initiative that works for other e-commerce platforms may not necessarily work for your specific context, and data-driven decision-making is key to navigating these differences.

• The Fast Buy initiative didn’t succeed, but provided us with valuable technical and user experience insights. We continued exploring alternative approaches for this initiative, and the learnings gained have helped refine future testing processes.

• Working with A/B testing is not as simple as it seems. Depending on the type of change, we encountered several setbacks that led to false negatives in the results. By introducing user interaction events at key stages, we were able to identify many implementation issues that would have otherwise gone unnoticed, helping us make more informed adjustments.

• Collecting feedback prior to the A/B test implementation was another major improvement. This allowed us to address potential issues early, preventing delays that would have impacted the test’s effectiveness. Previously, we needed at least one week of testing to gather meaningful insights, but with this feedback loop, we could gather significant data in just three days.