First off, we want to say we're extremely sorry for all the trouble the site and app has had this week. We received thousands of emails during the outages and there's no way to easily respond to everyone. If you contacted us to express your confusion or frustration, we're sorry we didn't get back with you. We spent that time working on getting the site back up and running.

The site went down because our database has been getting overloaded lately. Imagine each page load as someone walking through a turnstyle to enter a subway or stadium. Under normal circumstances the lines move through normally as everyone goes through one at a time. Now, imagine rush hour when there are many more people trying to go through those same set of turnstlyes. Things start to back up and people get impatient. In the web world, this leads to people refreshing the page which just makes the situation worse since each page load causes more and more requests to the database (essentially adding more and more people in line). This snowballs and eventually the wait time is so long and there are so many requests pending that the server doesn't know how to handle it so it starts throwing errors.

The main culprit is our Transactions table in the database. We have somewhere around 75 million transactions that our users have entered with somewhere around 50,000 new transactions added daily. This is a lot of data to constantly look up, add to and delete from. The hardware that we're currently running right now was built and set up around 2008 and the truth is, I think the site has gotten too big for our hardware to handle. Back then, we had outgrown several other hosting solutions and at the time, cloud based hosting was still in its infancy. We purchased hardware that we thought would last us for a long time, and has done so pretty well for the last 7 or so years.

For an immediate fix, we've backed up all transactions and are having you restore them when you log in by clicking a button at the top of the page. This process could take a few minutes but don't worry, your data is still secured and will be restored shortly (there are a lot of other people trying to do the same thing).

To help prevent these downtimes and database problems from happening in the future, we're working with an outside party to help us migrate into newer hosting solutions that aren't bound by a physical machine (namely, cloud based solutions that can expand with demand). This is the next logical step in our site's growth, but it's not something we want to jump into blindly. We're going to do our research and find the best solution to keep ClearCheckbook running smoothly for years to come.

If you're accessing the site through one of the mobile apps, you'll need to log into the website to manually restore your transactions for now. We'll work on updating this soon though.

Again, I can't express the depth of sorrow, stress and frustration I feel that we've had these issues. To compound on it, the 4th of July holiday weekend in the US has made it extremely hard to find outside help since everyone is out on vacation.

We'll keep you updated with the progress of the migration to newer hardware over the next week.

Brandon O'Brien
Founder of ClearCheckbook