One of the main things we’re working on at Chargify these days is improving the user experience, especially with regards to the speed of the site and the API. If you’ve been with us for a while, you’ve probably seen things slow down quite a bit over time. If you’re new, Chargify probably just seems slow most of the time. I’m here to tell you that we don’t find the current state acceptable, and we’re working on it.
Bottleneck
Our bottleneck is currently our database. We made a decision long ago to house our data inside a database system managed by our PCI-specialized host. This was a tremendous benefit at the time, and helped us achieve Level 1 PCI Compliance, which remains an important feature of Chargify to this day.
But, the benefits of being in this environment are now overshadowed by our inability to scale. We allowed ourselves to get behind the growth curve and now we’re trying to catch up.
Here’s what we’re doing to alleviate this bottleneck and other performance woes:
Query Optimizations
There’s been enough low hanging fruit in query optimizations that we’ve been able to maintain our performance through the growth of the past few months. But we’ve only been able to maintain, and not get ahead like we want. We’ve added indexes, optimized paths, and simplified logic.
Right now we’re working on a change to the way subscriptions are created that should give a substantial boost to the speed of signups. Over time, we’ve added features to the signup process (such as components, coupons, and taxes) which have made it a more and more complex path over time. Now we’re stepping back and approaching signup from a more educated position to make it as fast as possible. We’ll always be limited by the time of the external calls to the payment gateways, but we’re going to make the Chargify piece as fast as possible.
Caching and Feature Degradation
You may have noticed that the Site stats (revenue, signups, expired cards, etc), that used to be on every page in the UI, are now only on the dashboard. This is a feature that most people won’t miss being on every page, so it was an easy call to remove it and get a small performance bump.
Most of the values in those Site stats were cached values, but we realized our caching strategy for them could be better (especially the expired card count). Once we get a handle on some of the bigger performance hogs, we’ll be revisiting this and doing a better job of caching this piece, as well as a lot of the rarely-changing information in the interface.
API Rate Limiting
We first talked about API rate limiting a few months back, but tabled the discussion for a while based on feedback from merchants. What we heard was that we have to be very smart about the way we rate-limit, because no merchant wants a signup blocked in any case. Our goal is not to ever block a signup, but to make sure there are no API hogs (aware or runaway-script-unaware) affecting the experience for everyone.
We have the technology in place to begin rate limiting when we have to, but currently we’re collecting data about usage habits so we can put smart limits in place. When we do begin enforcing limits, these things will be true:
- You’ll receive warning before being throttled, and we’ll help you find ways to reduce your usage
- Higher paying plans will receive higher thresholds
- You’ll have time to make changes to your code, and we’ll be able to show you your recent usage for reference
New Queues
A lot of the work Chargify does is in the background: sending webhooks, sending emails, renewing subscriptions, etc. Our current queueing system (based on DelayedJob, for those curious) uses the app’s main database to store job information, only increasing the load on our main bottleneck. We’re mostly done converting these jobs over to a different solution (Redis/Resque) that should help alleviate some stress and keep our status site squeaky clean.
“Social Engineering”
Did you know we currently renew every active subscription ever created in Chargify, since Day 1? Sure, we separate the priority of “real” vs. “fake” subscriptions, but we try to keep doing work, without stopping, for every Chargify user. We’re currently planning ways of limiting the lifespan of test data. This will probably come down to asking you the merchant: are you still using this test data? If the answer is “No” or we receive no answer for long enough, we’ll prune your data or scale back your resources.
New Database Cluster
The biggest change we’re making is a move to a spacious new database cluster that we manage. It is still hosted inside of our PCI-compliant environment, but it will give us more control to scale and grow.
We’re on track to make this a New Year’s gift from us to you. We’re planning for very minimal downtime, and you’ll have ample heads up for when your particular migration is scheduled.
Conclusion: Our Merchants Rock
We are very fortunate to have merchants as awesome as all of you. We have many raving fans who are happy to spread the word about our subscription billing service. Some of these same fans have been quite up front with us though - they aren’t afraid to point out the Chargify performance issues and what it means for their business. They are also full of suggestions and offers to help, and for that we are grateful!
I wish this post was more about what we have already done and the results we’ve achieved. But that post will have to come later. You guys deserve to be informed, and I hope that’s what I’ve done. Thanks for all of your support, and I’ll be back soon with a results post.