Are You Prepared to Scale your app?

 

By Amir Schlezinger 

Are You Prepared to Scale your app?

 

By Amir Schlezinger 

A Game-Plan to Scaling in a Responsible Way

 
Scaling up your platform is a necessary part of the evolution of your company. Moreover, scaling should be something that’s always on your mind.
 

Only three-and-a-half years ago, Bigabid’s platform handled less than ten campaigns, with 5K QPS from one data source, under one data center. Today, we run hundreds of campaigns with roughly 1M QPS from four data sources, under three data centers!If scaling wasn’t constantly on our minds, we may not be where we are today. Here are a few tips to ensure that you’re prepared to scale, and that you’re set up for success when you do.

Questions to ask before starting – Before you attempt to scale, answer the following questions:

 
  1. What part of your system are you going to scale? Is there a surge of new users? Will an upcoming holiday drive a lot of new traffic your way? Did we recently deploy a resource-intense feature? Exploring these questions will help you determine which part of your system (if any) you should scale first.
  1. How much of the scale is predictable? You should always overshoot when preparing to scale. Will the platform need to handle twice the amount of traffic? If so, aim for even more than that. How much more will depend on how many resources (including money) you have for scaling.
  1. How frequently are you going to scale? You’d be wasting your time spending a few days every month working out how to scale your platform. You’d also be wasting your time putting a month’s worth of effort into making your platform scalable for the next three years. The way to properly address how frequently you’ll scale is by regularly considering the circumstances in which you’ll need to scale, so that you’re prepared to do so when the time comes.

    Case in point: At one point, each of our data centers had only one Redis instance running. We knew this would become an issue, but creating a cluster when it wasn’t yet 100% necessary would have been a redundant expense. We foresaw the day we’d need to upgrade (more on that below), and when that day came, we upgraded to a Redis cluster—but only one data center had enough traffic to call for a cluster. So, the other centers remain with a single Redis instance until their time for an upgrade comes. 

Metrics Are Your Best Friend

You need to be working with metrics before you scale, and after! In fact, metrics should always be on your mind. We could easily dedicate an entire article to metrics—how to use them to ensure that our platform is working well, and how to use them for scouting out problems before they happen. But for now, we’ll put it simply: if it has a pulse, put a metric on it. Here are some tips you should take into account regarding your performance metrics:

 
  1. Start with the basics. CPU, memory, network activity—for every instance you run, including managed instances such as databases.
  2. Monitor your scaling features. How you’re expecting to scale should not come as any surprise. With the proper metrics in place, you’ll be able to accurately foresee when scaling is needed.
  3. Every operation should be timed. You need to know how much time it takes to complete an operation. If you have a complex operation that’s comprised of multiple operations, you should measure each operation as well as the sum of them. Imagine having a process that loads a cache from a database while starting, and every interval updates the cache. In this scenario we definitely want to know how long it took to load or update the cache, but we’ll also want to measure how long it actually took to query the database. One timer is great for alerts, the other for investigating.
  4. Consider third-party components. If you’re integrated with third-party components, take into account that any one of them can break at any time, and plan accordingly.
 
Needless to say, metrics should be monitored and you’ll also need to set up proper alerts. When doing so, aim for general monitors. If you have a complex operation, you don’t need to monitor its individual components. If you receive an alert for a complex operation, it should guide you in the direction you need to investigate, which is where the metrics you put in place for the smaller components will come in very handy.

 

The Data Goldrush

Saving and analyzing data is the latest rage. You’re inclined to save every micro-action your user did or even thought about doing because… well, who knows? This might be the best predictor of user profitability.Unfortunately, saving too much data has become a common and costly issue. Data saved for “maybe one day” purposes more often than not winds up being unused data that becomes a burden, both in terms of expense (the bandwidth to save said data is costly in terms of server performance and the effort needed to sustain ETL processes), and in terms of HR managing it. Let’s take a step back; after all, we’re not Google. Be smart about the data you’re saving. Figure out if you’ll be able to use it before it goes stale. Being a hoarder won’t help, and with time you’ll feel increasingly uneasy deleting old data. So, best to avoid the issue before it happens.

 

Conclusion

Scaling is a lot about preparing, and being honest and realistic about our scaling goals. Indeed, if a team expects to scale successfully, the topic must be part of everyday conversation long before scaling actually happens. Only with proper preparation that comes from constantly discussing it, and from setting up the correct metrics and alerts, will you be able to ensure the success of your scale. Once you’ve laid the foundation for scaling, you should be sure to adopt proper habits for handling a scaled system, and have a plan in place for when something goes wrong. More on that in Are You Prepared to Scale Your Mobile App Part: Part 2.

 
If you’re ready to start scaling your app with Bigabid please click the “Contact Us” button below.
 

Results based advertising starts here.

How to deal with scaling your app on a day-to-day basis.

You’re now managing a scaled system. Congratulations! But wait—don’t throw your feet up just yet. You need to make sure you’re properly handling your scaled system, and that you have a plan in place for what you’ll do when something goes wrong (don’t worry; it’ll happen eventually!).

Habits You Should Adopt

One of the keys to scaling successfully is for scaling to be part of the daily conversation happening in your team. But saying you should be thinking and talking about scaling daily is easier than actually doing it.

With that in mind, here are some tips to keep you talking and thinking about scaling every day:

  1. Check your metrics daily. Ask yourself how the metrics you’re seeing today are different from the ones you saw yesterday, a week ago, a month ago, etc. Are you seeing a positive trend, or is there something that needs to be addressed before things start falling apart?
    Side note: Remember the Redis story from Part 1 of this article? Thanks to the fact that we were monitoring daily and alerted long before the issue became a pressing one, we were able to take an entire week to manage the upgrade, without any pressure. This well-thought-out execution relied on the regular monitoring we’d done beforehand.
  2. Anytime you deploy a new feature, ask yourself this question: How could this potentially break our system? It’ll help to examine which part(s) of your system the new feature interacts with, and consider the different ways in which it could affect your system with varying load.
  3. Aim for small, incremental releases. It’s much simpler to account for changes when you have an isolated environment for testing new features and seeing how they affect your platform.
  4. Don’t do it alone. Scaling should not be a one-man job. If you’re leading a team of developers, make sure you’re regularly discussing how their work might affect your platform’s ability to scale.
  5. Take note of your technical debts. You’ll inevitably accrue them, so be sure to write them down. Then, make sure to take a peek every so often to see whether now is the right time to address them.

What do I do when something goes wrong?

It’s crucial to prepare for the likelihood that something will go wrong. After all, it’s inevitable that at some point, some part of your system will fail. The good news is that you can prepare for that rainy day, and even mitigate some of the damage!

Here are tips for preparing for the moment when something goes wrong:

  1. Pinpoint what’s most important to you. If multiple fires suddenly start burning, you need to be clear on which one you want to put out first. Be clear about this before it happens.
  2. Define your system’s SLA. Understand what it is that could potentially kill your business, and what could be a mere nuisance. Does your system require 24/7 uptime? Is data integrity your top priority? Would an increase of one second in your response time create huge losses for your company?
  3. Define the parts of your system that scale easily, and the ones that don’t. By doing this, you’ll be able to tell which parts of your system need to be handled first. Some questions to consider:
  • Can you simply increase the number of servers you use? Pay a bit more and deal with the problem that way for now?
  • If a certain component fails, will that require manual intervention? Does everyone on my team know how to handle that scenario?
  • Do you have a single point of failure?
  • Are you taking a large financial hit by moving unmanaged services to a managed state?
  1. Have a rollback plan. Ask yourself how easy would it be to roll back this version, and what would happen if the data structure changes? Make sure you can revert back to your old schema and migrate new data to the old format, if possible, to prevent data loss.

Conclusion

For many, scaling is very intimidating—but it doesn’t need to be. If you’re well prepared, you can manage scaling with stride. Part of this involves accepting the fact that at some point, your operation will fail. Your system will crash. But there’s a bright side: you will improve and the process of scaling will become easier with time. Talk about it daily. Think about it always.

 
Contact us
Please fill out the form below to submit your interest.
Please review our privacy practices: read privacy policy.