Oracle Performance and Engineering

Today, in spite of having a roaring quarter Siebel took a nose-dive to close at $8.68. Noticeable stuff is Salesforce.com -its up and coming rival in the "on-demand" market - also lost one percent and change on the same day. From 2000 for about four years I was working in an on-demand software startup in Bay Area. I've had some personal observation to share regarding the performance and therefore, availability of this type.

As far as I remember, Larry Ellison probably was the first big guy in the industry to start talking about "Utility Computing". Utility computing was easy to understand - software will be provided like an utility. Just like you pay the bills to the local electric company for the Watts you use - you'd pay for the software without having to care for development, upgrade, or maintenance. The logic was (and, is) catchy. No one builds a powerplant at home to get electricity. Neither should one have a full-fledged development and maintenance for software if that's not the core business. "Utility Computing" was preceeded by "ASP" or, "Application Service Providers". Since then we really mention "hosted" (if engineers are talking among themselves); or "On-demand" if we're talking to potential customers (because IBM marketing demanded we do!).

Salesforce.com went one full step ahead and put "No Software" as its unilateral mantra, sometimes bringing attention to the fact that Dalai Lama doesn't need a software and, neither should we lesser mortals. In fact their US sales phone number is 1-800-NO-SOFTWARE.

Coming back to performance - I believe in the following characteristics regarding application performance -

1. 'Fast' or 'slow' is defined by the users, not by performance 'analysts' just the same way a stock price is dictated by the buyers and not by the 'analysts'.
   
   
2. The 'best' mathematical way of achieving 'fastness' is not doing it. Hey, nothing can beat 0 seconds! (well, till you...)
   Corrolary ONE: Often 'not doing it' is the best business solution as well.
   Corrolary TWO: The best solution is often unnoticed in production since it never manifests as a problem. Please remember this when you review the performance engineer.
   
   
3. When there's one performance issue, there are usually more. That's exactly the reason I didn't let my dad read my grade reports on a day he's tired or upset. People who believe in tracking and killing the "root of all evil" are perversely optimistic.
   

The whole 'on-demand' thing, IMHO, is just another delivery model. You eat pizza outside or order a delivery? Eventually it's really your choice. But remember for business they have to be extra careful when they home-deliver. If the pizza-guy is late, you have a right to not pay! I guess this is exactly where the 'on-demand' providers should be really careful.

Usually the characteristic #3 (see above) is what differentiates the 'on-demand' model from the others. You usually put a lot of customers on the same host (either app-server or database-server). So when even one single customer is causing a CPU spike thanks to his 12 million time-sheet records he needs to approve by month-end, all other 'innocent' customers put to share same resources as him suffer. Humans (and more so, enterprises) really really hate to be punished for crime someone else (sometimes, a competitor!) had commited. Just imagine PFIZER is asked to pay taxes on MERCK's income. They won't like it, will they? Similarly, try telling a customer that he couldn't log in for the entire yesterday because your biggest customer was "hogging" the entire database server because "this guy has 400 very complicated change orders he wanted to release". Well, good luck!

Business should be extremely aware of this. In the old client-server days you could send a "team" to the irate guy's office to "track and fix" the issue. With this new age "on-demand" your equivalent options would be to send a team to all your customers OR, fix it right there before it spreads to others. Difficult, yes?

Imagine a really 'cool' kid at your kid's birthday party motivating all other kids to jump and grab all the balloons from wherever they can! I bet it won't take very long for the kids to believe in him or her and follow the leadership. They're very serious in those matters. Same thing happens when a creepy performance or stability issue manifests in the 'on-demand' scenario.

The available resource is more limited now and the competition to grab it is much more intense than ever, but the 'heating up' of the system could cause it to go 'boink'.

That, not the marketing, is 'on-demand's biggest challenge that they must surmount. Very well planned resource-scheduling, fine degree of load-balancing, intense monitoring, gigantic load-tests with the realistic production data and set-up and above all a competent performance evangelist are truly needed by the business. Remember the customers do not need IT anymore. They have entrusted the provider with their data, process and application. But when one of them goes down all go down and you can never let that happen unless you really negotiated your severance well before.