Blogs To Read About DevOps, performance, and scalability:

1) Dev2Ops (Updated Every 1-2 Weeks) - As the name would suggest, Dev2Ops covers everything you might want to know on this subject, and features a lot of video as well. Much of their material is also positioned for people in devops roles in larger organizations, and how to manage release and control systems in large organizations which have different kinds of problems than close, small teams.

2) Atlassian (Updated 1-2x a week) - Atlassian’s excellent blog covers much of the infrastructure that teams need to stay in communication and on top of their business.

3) High Scalability (Updated 2-3x a week) - One of the best blogs covering topics around DevOps, High Scalability includes everything in their material from new ideas and thought leadership to practical examples and news coverage of everything happening in the world of performance and scalability.

4) MySQL Performance Blog (Updated 2-3x a week) - WIth a name like this, it’s hard to go wrong – You can’t really afford to skip this site at all if you’re responsible at all for database performance.

5) Code as Craft (Updated 3-5x a month) - By the Etsy team, Code as Craft describes the engineering and operations challenges and discoveries they’ve found at scale.

You can subscribe to all of these blogs (and our own) with one click now via our Google Reader Bundle! If you’re not a Google Reader user, OPML and RSS/Atom versions of the Bundle are also available so that you can use your favorite application.

Books! Here’s some great reading, whether you enjoy them digital or pulp, during your commute or at the gym, and anywhere in-between:

1) Web Operations: Keep The Data On Time by John Allspaw & Jesse Robbins. This is actually my current reading material and could almost be required reading material for anyone who wants to make a career in the development and operations world: John Allspaw and Jesse Robbins (with other select guest writers) break down some great engineering problems and stories about developing and scaling applications with a variety of tactics and issues at hand. If you only have the time or patience to read one book, this is the way to go.

2) Continuous Integration by Paul M Duvall, Steve Matyas, and Andrew Glover. The reasons why and benefits of CI are the subject of many other well-written blog posts (One well known example is here), and this is a great book on the topic with examples and details on how to get the most out of it for your organization.

3) The Art of Capacity Planning by John Allspaw. We’ve already talked about John above – This is one of his other books. John’s incredible experience in infrastructure and capacity planning includes notable roles at Friendster, Yahoo/Flickr, and Etsy.

4) Lean IT by Steven C Bell and Michael Orzen. “Lean” is a phrase that’s often used and and even more often misused, and so resources that help appreciate the value of what lean strategies are and aren’t are very useful. If effectiveness and efficiency are problems for your group, adapting more of the lean strategies will probably benefit you. Even if your organization is already successful with lean strategies, it’s always useful to see examples and case studies of

5) Kanban: Successful Evolutionary Change for Your Technology Business by David Anderson and Donald Reinertsen. If you’re part of a team and struggling as a group to manage and make progress on a number of projects simultaneously, you probably do work in an IT or DevOps organizing. They’re complicated and constantly full of different challenges at different levels, but there are resources available that can help you better plan and understand the work that you have to do.

6) This is a bonus one since it’s not directly about DevOps, but it is a critical book about management and business by one of the great legends of the topic. Management Challenges for the 21st Century by Peter Drucker is a critical book for anyone responsible for business goals and operational success. There’s not much more that you can say about it than that. 

What other blogs or books do you enjoy reading about web development and performance? What have you learned from? Let us know in the comments.

(Photo Credit: Talltim)

While every web stack is unique, when you start scaling there are often low-hanging fruit waiting to be plucked (and fixed) that eventually everyone will encounter. They may not be glamorous, but they are common — and easy to fix once diagnosed.

This week, we’ll break down two common classes of performance problems we see in typical web apps — one for ops engineers, one for developers — and general approaches to solving them.

The underprovisioned backend

Problem
You’re getting some publicity and seeing more traffic, but your app is getting slower, and you’re not sure why. You log into Tracelytics and see a graph like the one below. What does it mean? Why do requests seem to be spending so long in Apache or Nginx when you suspect it’s your app that’s slow? Shouldn’t they be spending all that time in the your app server backend (e.g., PHP, Rails/Unicorn, WSGI, etc)? Or maybe your app isn’t slow at all and there’s something wrong with your web server?

Diagnosis
That growing amount of “web server time” indicates your requests are spinning their heels queued up in front of your app server, waiting to be processed. Your backend can only handle a certain number of requests per second, and that rate is determined by two important factors: the average performance of individual requests, and the number of workers (threads, processes, server instances) provisioned for your backend.

A lot of teams spend a lot of time thinking about the former concern — writing clean, well-performing code, worrying about database indexes, etc — but then get bitten by the latter when they start to scale.

Worse yet, real-world user traffic is highly variable, making it hard to figure out whether the slow performance you were seeing during a traffic spike was more to blame on the code or was because of a misconfigured deployment or underprovisioned backend. One useful feature of Tracelytics in this scenario is our layer breakdown, which charts web server & app layer latencies over time to help you pin blame accordingly.

Solution
Of course, it’s always a combination of both factors, but for many teams with scaling challenges, often the simplest, lowest-hanging performance mistake they make is simply not provisioning workers fast enough to meet user demand. Do the math — if you have, for example, just 8 Rails Unicorn workers available, and know that on average your dynamic pages take ~200ms per request, then your deployment can support at best only 40 requests per second.

Without a well-provisioned backend, even if most of your pages would load right away under typical conditions, a sudden spike or even a slow-growing, unnoticed rise in demand could quickly eat up all available workers. Requests start to queue up in the web server or load balancer before they ever make it to the app layer, and users experience slow page load times or timeout errors.

It’s tricky to pick the right number of processes & threads that maximize concurrency on each app server without exceeding resource limits (CPU, memory). There are different operational concerns for every kind of backend and server environment — which are outside the scope of this article — but it’s not uncommon to find that the default configuration settings distributed with your server software are hugely inefficient without tuning.

But in the face of a 10x traffic spike, all that tuning won’t necessarily help if you’re not well-provisioned in anticipation already, and/or can’t quickly add capacity (more servers) when needed. Cloud services like Amazon’s EC2 Auto Scaling are a great way to schedule or trigger capacity increases and decreases to meet user demand. And don’t forget to exercise your stack with load testing, too — the screen shot above was actually taken from Dan’s post on load-testing the Reddit source.

The looped query

Problem
You wrote some code that looped over a set of related items — for example, a user’s comments or tags — and added a database query or service request inside the loop. Maybe you were working inside a loop in a template file, and just needed to grab a little more metadata inside that loop, so you called back to your app from the template.

You may have thought at the time: “But this query to grab the current tag’s last-updated date will return so quickly! It’s on a table with a great index! How could this be a performance problem later?” Or maybe you were following 37 Signals’ advice to “scale later” and just wanted to get it working quickly.

Diagnosis
At scale it’s not just about individual query performance: all those extra round-trips to the database can really hurt once you hit hundreds or thousands of items. Even if your database can fire back a response to each query in say, 1ms, the database adds a whole second of latency to run 1000 per-item lookup queries. Add in the time it takes your DB driver or ORM to format query arguments, send them over the network, wait for and process the response, and you could be adding several more seconds of unnecessary latency and extra work.

The screenshot above is taken from a trace of a WordPress page using a “comment karma” plugin that issues a SQL query for each comment. While each query takes only a millisecond or less, running 708 of them in sequence adds a couple of seconds of MySQL-related time, and the back-and-forth legwork in PHP to process the request & response adds more time, all for a total of four seconds to load all those comments!

Solution
Move that query out of the loop! Grab all the values that you need (e.g. all of a user’s tags, comments, or friends) with one big request, or in fixed-sized batches. Then once your app starts filling up with real data, you’ll be able to handle it gracefully. Double-check template code for per-item requests inside for loops.

Conclusion

Again, these issues aren’t glamorous, but we see them enough that they seemed like a good start to this series. In future posts we’ll look at more general classes of problems as well as specific pitfalls affecting certain web stacks.  And as always we’re curious to hear your performance stories, too!

(Title Photo From Quapan on Flickr)

The specific application we’re looking at is a message board (forum) that processes about 3600 requests per hour. Viewing the heatmap, we notice a pattern: there are basically 2 clusters of requests, those that take under 100 ms and then a set of slower requests that take over 600 ms, forming a line near the top:

We select the slow, high latency requests in the heat map and check out the top URLs:

Ah ha! This is a little more revealing to the source of our problem: The majority of these requests are related to “profile” pages, so investigating those seems like an appropriate place to start. Let’s break down an individual trace for the /user/profile/3.page request, where we notice that the majority of time is spent in a single SQL query:

Viewing that extent, Tracelytics provides us the exact SQL query being executed, along with the offending line of code in the application, in the form of a Java stack trace. The SQL query is:

SELECT COUNT(pm.privmsgs_to_userid) AS private_messages, u.*
    FROM jforum_users u
    LEFT JOIN jforum_privmsgs pm ON pm.privmsgs_type = 1
         AND pm.privmsgs_to_userid = u.user_id
    WHERE u.user_id = 3
    GROUP BY pm.privmsgs_to_userid

We connect to the MySQL database and run an “explain” on the query to provide us with the query plan:

mysql> explain SELECT COUNT(pm.privmsgs_to_userid) AS private_messages, u.*
    FROM jforum_users u
    LEFT JOIN jforum_privmsgs pm ON pm.privmsgs_type = 1
        AND pm.privmsgs_to_userid = u.user_id
    WHERE u.user_id = 3
    GROUP BY pm.privmsgs_to_userid \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: u
type: const
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 1
Extra: Using temporary; Using filesort
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: pm
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 2100274
Extra:
2 rows in set (0.00 sec)

In the above ‘explain’, we can immediately see the source of the problem: the “type: ALL” under “table: pm” means no index is being used when accessing the private_messages table and MySQL is doing a full table scan.

For any table with a very small number of rows, this probably wouldn’t matter. Unfortunately, this particular table has over 2.1 million rows and MySQL indicates it is going to examine them all (“rows: 2100274″). This is likely the source of our performance problem. To verify, we look at the table’s DDL with “show create table” and discover that the table actually has no indexes at all. This is the exact type of issue that only becomes a problem at scale: When the message board was young and only had a few users, this page was fast and seemed innocuous. Now that it’s part of a big database and there’s a lot of data being stored, even if the overall level of traffic is the same over time, this huge database takes a much longer time to scan through. Let’s say that one more time: Not every problem of application scaling is the result of increased users or load – Your storage and methods of accessing that storage can break down, too!

Let’s solve this issue quickly: By adding an index on the the two join conditions (privmsgs_type and privmsgs_to_userid), the query should become much more efficient.

We run:

alter table jforum_privmsgs add key privmsgs(privmsgs_type,privmsgs_to_userid);

By running this command, we’ve added an index to the table that the query can run against. To verify it is actually being used with our query, we run the same “explain” again:

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: u
type: const
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 1
Extra: Using temporary; Using filesort
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: pm
type: ref
possible_keys: privmsgs
key: privmsgs
key_len: 5
ref: const,const
rows: 8
Extra: Using index
2 rows in set (0.00 sec)

The “key: privmsgs” output indicates that our new index is being used. As a result, the number of rows examined on the jforum_privmsgs table has dropped from over 2.1 million to just 8 rows. We’re back to the small table sizes that are very fast for MySQL to scan through.

About a half hour later, we take another look at the heatmap. There is a very noticeable change around 2:15, right after we added the index:

We filtered our traces down to the /user/profile/3.page URL to see the effect our change has had on that particular page:

The profile requests are now averaging 11 ms. Originally, they were around 640 ms, which makes this about a 58x improvement. We made a very small change that provided a very large impact on the performance of our application! Not every problem will be this straightforward to solve, but without complete information about the time spent in each part of this request, this would have been a much more difficult and complicated problem to diagnose.

You can get this same performance data for your applications by signing up for a Tracelytics trial today: Start tracing today!

Friends, customers, and co-workers from all over came over to help us break in the new space, and play some Foosball on our newly finished table.

The Tracelytics logo has never looked so delicious.

Thanks to everyone who came and helped make this a special night for us. We’re glad the rain didn’t stop people from coming by to visit! If you happened to miss us yesterday, feel free to drop by our office and see what we’re up to and what new tools we’re developing.

- The Tracelytics Team

If you want to check out what we were doing there, here’s the recording of Spiros presenting on Tracelytics for the panel.

Here’s a few other startups that excited us:

MemSQL

You might already be convinced that you need to keep your DB hot set in memory, especially if you’re in a virtualized/cloud environment. So why not use a database that assumes that case and is optimized for it? That’s the premise behind MemSQL, a YC2011 graduate that won the audience award for databases at UTR. They boast 30x faster than disk-backed MySQL and a smaller multiple over ramdisk-backed MySQL. You might be thinking: “isn’t this just VoltDB?” Well, there’s two big differences: first, it’s mysql wire-protocol compatible, meaning no pre-writing stored procedures and that it will be drop-in for many customers (MemSQL compiles your queries and caches them automatically). Second, it isn’t in GA yet, so there’s no way for you to try it out. This is one we’ll be watching going forward!

ScaleArc

At some point in every engineer’s life, they’ve considered writing a SQL proxy. Whether it’s to automate load balancing, sharding, or gaining visibility into the performance of your database layer, a proxy that speaks your SQL flavor down to the wire protocol would go a long way to make those tasks a breeze. That’s what ScaleArc gives you. This might not be as sexy as those new-fangled database technologies flying around (see above), but it solves a few problems, like caching and profiling, without any code modification.  Particularly useful if you don’t have full control over the app.  It’s also an example of a developer-driven product, and we have a lot of respect for that here at Tracelytics.

DataDog

Have you seen the Velocity 2010 talk “A Day in the Life of Facebook Operations”? I had a craving for a dashboard like the one they use internally to keep track of operations events and code deploys. At the same time, on the other side of the country, DataDog was working on a collaborative ops vision very similar. Their service integrates feeds from dozens of monitoring sources (Ganglia, Nagios, GitHub, etc) and ties it together with real-time graphs and social commenting/sharing features for internal productivity.

(Full Disclosure: Tracelytics and DataDog are both funded by Battery Ventures)

And Finally, Your Moment of Zen

Though all the panels at UTR were full of insightful, hard-hitting judges, few were more entertaining than Joyent’s Jason Hoffman:

Under our Under the Radar Radar?

What did you enjoy at UTR?  Something we missed?  Let us know below!

We’ve just settled into our new office space on South Street in Boston, and we’d like to invite you all to a party at our offices to break them in. If you’d like to come, please feel free to RSVP here on evite so that we have a rough headcount for how many are coming, and please feel free to bring friends or colleagues – The more the merrier!

When: May 1st, 5:30 PM and on!

Where: 133 South Street, Boston MA (One block from South Station)

RSVP Here: Evite Page

The latency histograms on our overview page give a nicely detailed alternative to the “average” latency trendline they accompany, showing underlying components to the average response time.  The concise representation may fool you, but these once were the slowest feature in Tracelytics, sometimes taking over two minutes to load!  This obviously needed to be fixed, but it’s hard to know where to start working optimization when you need to make something not just 10% faster, but 1000% faster.  I faced a number of challenges, starting with algorithmic complexity and ending with fine-tuning python code.

Histogram 101

First, a quick review for context: a histogram is a plot of the frequency of observed values in a given sample.  In our case, we’re showing the distribution latencies for requests to a web app.  It’s usually drawn as bar charts due to the discrete nature of the bucketization.  We draw a continuous-looking line chart due to the fine granularity of the data:

What goes into building this?  Well, to draw the chart we need to know how many traces took 1-2ms, how many took 2-3ms, 3-4ms… all the way up.  And the buckets should be sized intelligently based on the range of the data.  Which means that in the most naive solution, you’re looking at data proportional in size to the number of traces that have been observed.  Ok for small data sets, not so ok for larger ones.

Where did I start?

We knew (thanks to the traceview) a lot of time was being wasted fetching millions of traces worth of latency information from MySQL. Gathering this much data would even be slow locally, so the trouble doesn’t end  at the database, because we would still need to sort and analyze all of the data, only to condense it down to a couple of kilobytes to send off to the user. That’s too time consuming and requires too much work by several orders of magnitude. From that conclusion, we decided the best course of action was to find a way to drastically reduce both the amount of data we had to search and the amount of data we had to retrieve.

Fortunately, TR and Dan had already done some of the thinking for me. The solution we found was a variation of a technique called quantile summarization, used in financial analysis, which allows for arbitrary quantiles to be queried over a sliding time window. A small amount of information, called a “quantile sketch,” is stored for the latency distribution over a time slice.  These sketches can later be merged with sketches of other time slices to get information regarding an arbitrary time window.  (Want more details?  Starting point: “Continuously Maintaining Quantile Summaries of the Most Recent N Elements over a Data Stream“)

The end result is a much faster query of the database and a lot less information being passed over the wires. However as a trade-off, the data requires a significant amount of post-processing–the merging of the sketches–to be useful. I whipped up an algorithm for this post-processing in Python and gave it a whirl and…

…the graphs still took about 30 seconds to produce! I couldn’t believe it. I opened up a traceview, and looked for a gigantic chunk marking the amount of time spent in the database. But as it turns out, the database latency was only a thin sliver. Our plan to speed up the database leg of the journey was a huge success, but my simple Python code was taking nearly half a minute to run. I was already using the optimal algorithm from the papers. I couldn’t even remember the last time I had written something so slow. Frankly, I was embarrassed.

Well, then what the $%* is going on inside Python?

Have you ever wondered how Python handles all those fancy list operations? I certainly have, and like most people, I have no clue. Looking at my code I couldn’t immediately see an obvious problem that was causing things to crawl.

After I turned on cProfile support in Tracelytics, we built up a few traces to see what evidence Tracelytics could share on itself. According to the data, the problem stemmed from 20 seconds wasted in thousands of calls to Python’s pop method. Why would this happen? Shouldn’t popping from the ends of a list be fast?

As it turns out, popping an item from the front of a list is a very bad mistake because it automatically shifts all the remaining entries forward one slot in the array.  You could look at code like me, or maybe save some time with this handy entry on the Python wiki I found later.  Keep in mind that we weren’t using a deque and pop(0) is very similar to remove.  By making this mistake, I had accidentally made my linear algorithm a quadratic algorithm. This is exactly the type of silly mistake I would have never noticed without profiling. With the continued help of profiling, I was able to make other little tweaks with memory allocation and method calls that resulted in shrinking the post-processing time down to a palatable sub-second latency.  Here’s a traceview of the work done generating a typical latency histogram:

The entire trip from gathering the request, seeking the data, processing, and returning the data now maxes out at a few seconds. I’d say that is certainly a fair improvement from two minutes by anyone’s standard.  If a fresh intern could speed up something in the Tracelytics web app 100 fold, imagine what you can do to your app!

Layer Latency

Latency is one of the most important metrics available for keeping tabs on your website. Not only does it affect the user experience, but it can also serve as an early warning for lower-level operational issues. However, setting a single alert on a complex system is very unreliable — setting a high upper bound provides good protection against catastrophic failure, but what about finer-grained visibility? MySQL has a completely different latency distribution than Apache, and their failure modes can be quite different.

That’s why we are adding layer-based latency alerting. Now you can set alerts on individual layers within your apps. Is memcache evicting too many keys, causing an abundance of slower SETs? Has the data set in MySQL grown beyond memory, causing queries to slow down or time out? Are there queueing problems in Nginx, caused by an influx of users? Layer alerts will alert you to performance issues as they develop, so that you can get in front of them right away.

Error Rates

Another big category we’re adding is error rates. In an ideal world, your webapp would never encounter an error. Unfortunately, connections fail, APIs get overloaded, and developers even occasionally forget to handle edge cases (shocking, but true). In a complex app, error rates are one of the most important metrics to keep close tabs on. Tracelytics will look for errors anywhere in your app, and report them if they become too numerous. This means we’ll monitor everything from DB connection errors to 500s from your webserver, and let you know if something looks amiss. We’ll even email you a list of the most common, recent errors we’ve seen, so you’ll know exactly where to jump in to fix it.

Host Metrics

While it’s great to take the 10,000ft view of your application, sometimes you just need to dive down and know what’s going on at the bare (or virtualized) metal. Tracelytics already lets you visualize host metrics right in the context of your applications performance, so now we’re taking it one step further and adding alerts on those host metrics. We’ll notify you when any metric exceeds your configured thresholds, making sure you’re up to date on exactly what’s happening.

Our Guest Today: LAMP

You may be familiar with this acronym: Linux, Apache, MySQL, and PHP. Of course, Tracelytics works with many different stacks, but LAMP is a very common web application engine. Here’s what the stack looks like, assuming that your database is running on a separate host, either physically or virtually:

Hosts + Layers = Apps

Before we get our hands dirty, let’s take a moment to break down the LAMP stack into various parts.

• Hosts: Web applications are software running on machines, be they physical or virtual. In this example, the Hosts are alice and bob. Hosts form the underpinning of performance with the resources they provide for the app to consume.
• Layers: Each component that is involved in servicing requests for your app is a Layer of the stack. In this example, Apache, PHP, and MySQL are each considered Layers. Typically, you might think of each service that runs in a distinct process as a different Layer.
• Apps: Finally, your entire production LAMP stack is a logical grouping of software components and hosts that work together to serve requests. Let’s call this an App. In the simple LAMP case, everything is considered part of the same App. If you have a separate staging environment, that’s a separate App.

How Does Tracelytics Work?

How does Tracelytics gather data about Hosts, Layers, and Apps? Each is handled slightly differently.  The following diagram illustrates instrumentation points in green.  Explanation follows!

• Hosts: You want to know whether your app is starved for resources or overprovisioned.  For this reason, the Tracelyzer agent, installed on each host,  gathers metrics about utilization of CPU, memory, I/O, etc.  The Tracelyzer is a lightweight daemon, provided as an APT or YUM package, that you can install on every host in your deployment that you want to monitor.
• Layers: Each Layer is presented slightly differently by Tracelytics depending on what features are interesting: for example, when considering the MySQL Layer we are probably interested in queries, tables, and databases, and when looking at Apache we will want to see response codes and HTTP methods. Tracelytics follows requests through your stack, starting at the top layer, which is usually some kind of webserver. To get visibility into the request starting at the web server, you install instrumentation in the form of a plugin module (eg. Apache or Nginx module). In the LAMP stack, this instrumentation watches for when requests arrive into Apache from clients, when Apache passes along the request to PHP, when PHP replies with content, and when Apache finally passes that back to the end-user.
• What about getting more detail about MySQL and PHP? You can go even deeper, if you install instrumentation for the next layer of the stack down: PHP. That instrumentation, provided as a PHP extension, listens for incoming requests (e.g. when Apache hands them off), records interesting events such as calls to MySQL, errors, and more, and then notes when PHP finished processing the request and returned control to Apache. Because PHP is watching what queries it executes, there’s no need to run a modified MySQL server.
• Apps: An app is a logical grouping of Layers running on Hosts. It’s defined by you, the user, to help you keep track of everything you care about. By default, all new layers are added to an app called “default.” User-created apps are defined by their entry point; The point at which requests enter the stack. In our example, that is Apache running on alice. After you have defined the entry points for an app, Tracelytics will follow requests passing through the entry point and automatically populate the remaining hosts and layers in your stack. Because the discovery happens automatically, Tracelytics understands when multiple Apps are sharing Host or Layer resources (eg. a database). A Tracelytics account can support an unlimited number of apps, which can be useful to segment your performance data across different environments.

Solving Performance Problems

All of this information comes together in Tracelytics to help you solve web application performance problems. Here’s a few examples of finding problems in a LAMP stack.

• Request Queuing: One common tuning problem is getting the correct number of worker processes/threads in the application layer.  When this number is too low, it can causes requests to queue in the webserver, a bottleneck adding latency to every request.  Because Tracelytics starts monitoring requests at the web server level, it’s easy to catch this problem–if too much time is being spent in Apache before the request is handed off to PHP, this is the likely culprit.
  
  In the picture above, we can see that a significant amount of time was spent in the webserver before it was able to pass the request to a worker thread at the app layer.
• Bottlenecks: What’s the worst leg of the critical path for a given request?  We watch every query made in each trace, which lets Tracelytics show you the low-hanging fruit in terms of slowest queries, frequently-executed queries, and queries that keep users waiting the longest overall. And the same for remote service calls, methods in your app, and more!
  
• Query Loops: It’s generally faster to retrieve a set of rows from a table all at once rather than making a separate query roundtrip for each. However, an unfortunately common design pattern is to fetch a list of items, then execute some other query (eg. to get an attribute of the item) once for each. This can be hard to catch if you’re just looking at slow query logs, because generally each individual query is relatively quick. But they start to add up. Because Tracelytics is watching each called to request makes, it’s easy to isolate and optimize query loops.
  
• Underprovisioning: Easy juxtaposition of latency graphs and host-level metrics allows you to quickly determine if bad latency is related to lack of machine resources.  That’s why we pull in host data along with tracing data:
  
  Look at how the latency rise coincides with increased memory pressure and CPU utilization– Perhaps you just diagnosed a memory leak?

Let us know if you have questions or comments about this, or other ideas about ways you could use these tools to diagnose performance issues in your application.

Here’s a few ways that Tracelytics is ahead of the pack, a bit about why we raised this money, and where it’s going to take us. Hang on!

Smart Tracing

Beyond Monolithic

The modern web application spans layers, hosts, and applications–why settle for partial and piecemeal visibility into your stack? Tracelytics is the only application performance management solution that provides full-stack visibility, following requests across boundaries between layers, hosts, and applications. Our smart tracing goes the extra mile so that you can solve tough problems that would be overlooked by legacy solutions. We are now able to use our smart tracing in PHP, Python, and Ruby, and we’re working on more for the future.

Beyond Averages

Traditional APM figures out which questions you might ask and presents the minimum data necessary to answer those, often in the form of averages you wish you could break down just a bit farther. Our innovative heatmaps and data-sifting features help answer questions that you–and we–haven’t even thought of yet. We’ve now analyzed over 500 million individual traces sent to us by our customers and users since launching the software, and helped them understand what that data really means.

Beyond Per-Host Pricing

Are you stuck paying per-host to get visibility into your apps, even in the autoscaling world of the cloud?  Tracelytics is priced based on the configurable volume of requests you decide to trace—not a legacy per-host model. Install everywhere, get visibility into your entire infrastructure, and pay only for what you use. With Tracelytics, you won’t have to cut corners to keep your costs down.

New Free 30-Day Premium Plan Trial

We’re also launching our new 30-day free trial, letting you experience the best of Tracelytics with full access to the Premium plan. Try it out today–no credit card is required and you’ll be up and tracing in minutes. If you’re ready to start really understanding what makes your web application tick and how to improve it, you can sign up now, or watch a 2-minute install walkthrough. It’s really that fast.

What’s Next

We’ve raised this money for two reasons: to keep improving the most comprehensive and powerful full-stack APM solution, and to scale our sales and marketing functions as a business.  In support of this growth, the company, currently located in Providence, RI, is opening a headquarters in Boston.

We’re also hiring, and already we are welcoming two awesome new team members: say hello to Sarah Ducas and Brian Whalley!  Could you be the next tracer?  Check out our jobs page.

And last but certainly not least: we’ll also be sharing a number of new innovations and features in the coming months—stay tuned!

Our Investors and Allies

Bain Capital Ventures joins our existing investors Google Ventures, Battery Ventures, and Flybridge Capital Partners in backing Tracelytics. Bain brings to the table deep knowledge of the application performance management space as well as a hands-on approach that has impressed us from the beginning. We’re also supported by a heavenly choir of angel investors and, of course, our customers and fans.

Together, we are making the web a faster and more reliable place.

There’s therefore a strong incentive to maintain a record of all operations events (releases, configuration changes, upgrades) that can be easily correlated with performance and error data to quickly resolve problems and find root cause for issues.  That’s why Tracelytics provides an API for recording operations events.

In this blog post, I’ll introduce you to using our command-line tool tlog to annotate events in Tracelytics, and then go through a quick case study of how we (and our customers) use it to track regressions in code.

Introducing tlog

The tlog tool is a command-line client for the API that allows you to report system-wide events that appear as annotations above the charts shown in Tracelytics.   It is automatically installed as part of the tracelyzer package, so you can run tlog wherever you are tracing.

$ tlog --help
Usage: tlog [options] [MESSAGE]
Logs an annotation message for viewing in Tracelytics.

Options:
  -h, --help   show this help message and exit
  -m MESSAGE   specifies the message to report.
  -a APPNAME   specifies an app to which this message belongs. (optional)
  -H HOSTNAME  specifies an alternate hostname associated with this
                   message. (optional)

It will then show up like this on your charts and heatmaps (though collapsed by default):

It’s that simple!  Here’s the full tlog documentation.

Integrating annotations with operations processes

Of course, for these ops annotations to be useful they have to exist! That’s why we provide the command line tlog client that’s easy to integrate with your deployment and operations processes. The most common way this is used is to note new code deployments. Here’s a screenshot from a great blog post written by SeatGeek:

You can see here that a performance issue was resolved by a subsequent deployment of code.

Okay, great, so how would I do that?  For a basic approach, you’d just have your deploy scripts execute something along the lines of:

$ tlog -m "Pushed a new release." -a "My_AppName"

For a more interesting example, which integrates the git branch and commit hash, here’s a fabric snippet that we use in our deploy automation:

def _tlog_deploy(app, extra=None):
    user = local("whoami")
    host = run("hostname")
    with cd("/deploy/tracelons/tracelytics"):
        branch = run("git branch | grep '*' | cut -d' ' -f2")
        commit = run("git log | head -n1 | cut -d' ' -f2")
    if extra:
        extra = ' ' + extra
    else:
        extra = ''

    with settings(warn_only=True):
        run('tlog -m "[deploy] %s deployed %s (%s) on %s%s" -a "%s"'
                % (user, branch, commit, host, extra, app))

In practice: finding a performance regression

It’s always great eat your own dog food, even if sometimes it’s embarrassing to get caught eating dog food.  So, here’s just one incident in which we used Tracelytics to ensure that Tracelytics was running well.

After noting that one of our chart endpoints was suddenly causing latency alerts, we investigated in Tracelytics and  noticed that requests to our web app were spending more time in the MySQL layer than they had previously:

early warning of a performance issue!

And it turned out that this coincided with a code release—how unexpected!  Because we do very frequent releases inspired by continuous deployment, we already knew where to look.  In fact, a re-factored query had lost a constraint on a subquery which caused it to be doing work that was later thrown away by app logic.

So we were able to quickly find the issue, but as usual there’s more than one way to skin a cat.  In this case we were able to verify the problem using Tracelytics before even looking at the commits and code involved. Drilling down on the MySQL layer, we were led to a particular recently slow query,  as well as traces showing the circumstances under which it manifested:

Fortunately, in addition to being easy to track down, the issue was also easy to resolve!

Beyond tlog

Looking to get the most out of Tracelytics?  This isn’t the only way to extend your visibility: check out last week’s post on framework instrumentation, or head to the documentation to read about the custom event API.

Drupal

The Drupal module for Tracelytics captures the currently active menu item and source file (what might be considered Drupal’s equivalent to actions and controllers) as well as optionally partitioning traffic into different categories based on whether the user is anonymous, logged in, administrator, or whether it’s actually an action running from shell or cron.  Finally, all of this is configurable through Drupal’s admin system.

It’s under development on drupal.org — check it out!

ExpressionEngine

If you’re running ExpressionEngine, you can get improved reporting of controller and action information by using Ian Cook’s EE hooks for tracing. They actually have New Relic hooks built in as well for comparison. The code snippet, as well as more background, is available here.

Other frameworks

Of course, Drupal and Expression Engine aren’t the only frameworks in the world—far from it! We also have CodeIgniter hooks documented, as well as modules for several Python and Ruby frameworks (see support.tracelytics.com).  But what if you are using something else? Read on to find out how simple it is to add custom tracing data with Tracelytics.

Custom instrumentation API

There’s a lot of things Tracelytics can tell you about your app right out-of-the-box—what errors are happening, what URLs are slow, which queries could use some optimization, etc. But answering some questions, like “what controller and action am I in?” or ” is this a logged in session?” requires a bit of insight into the actual application logic. Using our custom instrumentation API, it’s super easy to add extra information to your traces in semantically meaningful ways.

There’s documentation available on our support site, but here’s a quick concrete example. We’ve mentioned controllers and actions several times in this post—these are great to annotate on traces so that Tracelytics can group requests intelligently, and they are super easy to add even if you are running a custom framework. Simply call oboe_log (or the equivalent in Python/Ruby) with an associative array containing the keys Controller and Action, like so:

oboe_log("info", array("Controller" => $controller,
                       "Action" => $action));

…which will give you a new way to break up and drill down on your data:

Of course there’s tons more you can do—segment API traffic and real-user traffic, log errors on 404s, profile heavy parts of your application code—check out the documentation for some ideas.  And let us know what you come up with; we’re happy to hook you up with some free tracing in exchange!

The problem

Since performance data is complex and there are many ways to examine it, we wanted to make sure you are getting the most out of the Tracelytics interface.  Some apps tackle the problem with persistent contextual help paired with key page elements (we’ve tried that). Other apps use offsite documentation where they can elaborate on use cases (we have that too).  Still others use more conversational documentation such as blogs to call attention to features (hopefully why you’re reading this).

What we decided to do

Today we are announcing the Tracelytics Page Guide. This is a dynamic in-page contextual help system that we  use to call attention to important page features, use cases, tips & tricks and anything else to help you reach insightful understanding of your data as fast as possible. Unlike persistent help (those little ? icons you see in some apps), the page guide doesn’t get in your way after you’ve learned about the features of a page. But it’s always waiting for you on the sidelines, just in case you need a refresher. Unlike documentation sites and manuals, it is attached to page elements so there is no confusing where the feature is located. Finally, the Page Guide is dynamic: the items in the guide depend on what features you see on the screen.

How does it work?

Pinned to the right side of your browser window, on pages with features and elements we think you should know about, you will see the Page Guide icon.

Click on the Page Guide icon and you will see numbered flags get attached to various elements on the page.

We highlight the element that the Page Guide description is about. Clicking on any of the numbered flags updates the description at the bottom of the screen with info on the item you’re interested in.

You can also scroll through the Page Guide items using the previous and next arrows in the description area, providing a tour-like experience:

You won’t see the Page Guide on all pages in the app–currently, it’s applied to a few representative pages–but that doesn’t mean we’re done!  Feel free to email us, tweet us, or jump in our IRC channel at #tracelytics on freenode and let us know what pages or features we should make sure to get into the guide in the coming days.

One more thing…

We think that pageguide.js could help a lot of other apps and sites so we’re thinking about open sourcing it.  Just like feature requests, email, tweet, or IRC if you think this is a good idea, want to contribute, or just want to say hi!

There’s also a few new features we’ve introduced and changes we’ll be rolling out over the next few weeks.  Here’s a quick overview:

Trace Cap Enforcement

We price our service per-trace because we believe your visibility shouldn’t be limited by per-host fees, but rather provided on a pay-as-you-go basis–just like cloud infrastructure. Deploy the Tracelytics agents across your entire infrastructure and only pay for the actual volume of traces you consume.

Beginning March 15th, we’ll be enforcing limits on traces per month according to the plan you’re subscribed to.  This means that after your monthly allotment of traces, plus a grace window, your account will stop processing new data until the end of your billing month.  You won’t be cut off from the data–you will still have access to the traces processed up until the cutoff–but the data will no longer be live.

To help make this a smooth transition, we’re providing several new ways to be updated on trace consumption.  A meter in the app will keep track of traces used, and you’ll be automatically warned (via email) if you’re getting close to the limit or exceeding it.  Finally, as mentioned above, this is a soft limit: we’ll process a few extra traces for users that are only slightly over.

Email as Login

No more remembering service-specific usernames!  Just use the email you registered with to log in to your Tracelytics account(s).

Alerts

You don’t need to be logged into Tracelytics every minute of the day to stay on top of your site’s performance.  With alerts, set up latency constraints and be notified when your app is in violation.  We’ll take care of the math to make sure it’s not just a fluke triggering the alert.  Coming soon: alerting on machine metrics!

Partitions

Ever wondered how your app performs for logged in vs logged out users?  Browsers vs web crawlers?  User type A vs user type B?  With partitions, you can segment data in Tracelytics to reflect the populations you want to get insight on.

And Lastly: A Big Thanks!

It’s important to thank our beta and early-adopter users who helped make Tracelytics what it is today–you guys rock!  The feedback isn’t over; we’re always looking to hear your thoughts.  Get in touch with us at contact@tracelytics.com, on our support site, or chat with us in IRC on freenode at #tracelytics.

Tracelytics Alerts are similar to other monitoring solutions, with one important difference. Instead of keeping track of machine availability, network connectivity, or other low-level metrics, Alerts measures what matters most: your website’s performance. Performance is more that just latency distributions and sparklines; it is a direct measure of how effectively you are serving your content to users. Because of this, measuring, tracking, and keeping you up-to-date on your website’s responsiveness is a more relevant metric of availability than availability itself.

Why Performance Monitoring?

The reason that performance monitoring is so powerful is simple: high latencies are a leading indicator for downtime and other serious operational issues. Before a machine goes down, either due to a traffic increase, hardware defects, or network contention, it will generally start to fail by responding to existing requests more slowly. For example, a RAID5 array with a bad disk will not stop working; it will simply respond to requests more slowly, due to having to recompute the lost data from checksum blocks. Unless you’ve had the forsight to set up OS-level monitoring on every critical device on your system, this sort of problem could easily go unnoticed. However, setting up a single top-level Alert can warn you of lurking issues before they become problems.

A latency spike, then 4 hours of downtime. Not good!

Slicing and Dicing

Because it’s so important for Alerts to operate on data you already work with, we’ve built them right into our Layer Summary page, at the App level. Apps tend to be pretty coarsely grained, so Alerts also support filters on domains, URLs, controllers, and actions. Once you’ve picked a threshold and verified its behavior, we’ll keep an eye on it. By default, we’ll email you when something goes wrong, and again when the system recovers. And you can tweak them whenever you like.

Review and Drilldown

Of course, learning about a problem is only the beginning of fixing it. Back on the Layer Summary page, we’ve added a new section devoted to reviewing past Alerts. This shows only existing Alerts that could be triggered by the data on the page. This lets you see your past Alerts, right next to the data that caused them. From here, you can drill down to the offending layer, find the machine that’s going bad, and make it right at quickly as possible.

A note for the mathematically inclined

As you review past Alerts, you may notice that the data in some charts may seem to disagree with when we saw a violation of your Alert. This is because they are calculated slightly differently. Most data is averaged over the displayed window — points on the day view are 15 minute averages. To provide up-to-the-minute responsiveness, Alerts (including previews) uses exponentially weighted moving averages. This (honestly, fairly boring and standard) way of calculating the current value of a quantity smooths over spikes and noise in the data, while still providing a fast response to meaningful changes.

Get Started!

We’ve already gotten a lot of mileage out of these internally, and we look forward to hearing your feedback. If you think you could benefit from monitoring at a higher level, set up a few Alerts, and let us know what you think!

What is this meant for? To help you get started, here’s a few places we’ve found partitions to be useful:

Logged in vs. Logged out

First and foremost, Tracelytics shows you how fast the various contents of your page load, which directly translates to your user’s experience on your site. Frequently, though, not all users see the same content; specifically, logged in users tend to have a much richer experience. That richness comes at a price, though.

If your pages typically load in 0.8 seconds, is that because everybody is getting a good experience, or is your barebones landing page hiding the fact that most registered users are actually waiting 4 seconds before all their content shows up? With partitions, you can divide up your traffic and make sure that even both groups are seeing the kind of performance you want them to see.  (And it’s true–this is important. Just ask Stack Overflow.)

API vs. Browser vs. Googlebot

As businesses grow, they frequently run more than just a website. There will be internal tools for both business and development, an API for 3rd-party developers, a beta site for adventurous customers, and more–often all running from the same codebase on the same hardware. Code and infrastructure reuse is great, but it can make it difficult to understand what kind of experience each group has.

With partitions, you can tag where each request is coming from. That information is propagated to all layers within that app, so even if only your load-balancer knows that the incoming request is for the beta site, you can still view all database queries the beta site incurred (and hopefully see that they’re better than what you have in prod right now!). Similarly, web crawlers tend to disproportionately visit slow or uninteresting corners of your web site. By separating these requests into their own partition, you can attack and optimize the pages that are slow for real, live users. Partitioning like this ensures that no matter how your users visit your site, they’ll get a fast, consistent experience.

Semantic Separation

One of the larger chunks of code we have is our trace-processing pipeline. In order to ensure that all our customers get their data in realtime, it’s important that all the machinery is well-behaved and consistent. All of our customers run different stacks and configurations, and, as a result, their traces turn out a little bit differently. We use partitions to categorize each trace by its dominant layer type, e.g. SQL, services, or profiles. Since we process each of these types in separate modules, they each perform a little differently. Being able to split them up in the same way that we think and talk about them makes debugging and performance tuning that much faster.

Get to it!

Hopefully by now, you’ve thought of some way you could divide up your own apps with partitions. If so, head on over to the partition documentation, get started, and let us know how it goes!

The big problem we had after writing load tests was that timing data gathered by multi-mechanize is inherently external to the application. This means it can tell us the response times of requests when the app is under load but doesn’t identify bottlenecks or configuration problems. So we need to be gathering a bit more data about how the internals of our web application respond to the workload.

For this blog post, I’ll be using Tracelytics’ instrumentation, which is installable from OS-native packages and, in the case of Reddit, takes care of instrumenting nginx, pylons, SQL queries, memcache calls, and Cassandra calls automatically.

Test 1: ramp-up read threads

[global]
run_time: 1800
rampup: 1800
results_ts_interval: 20
progress_bar: on
console_logging: off
xml_report: on
[user_group-1]
threads: 10
script: view_base.py

[user_group-2]
threads: 10
script: view_all.py

[user_group-3]
threads: 10
script: view_all_new.py

[user_group-4]
threads: 10
script: view_all_top.py

So, this test is going to run for 30 minutes and generate steadily increasing read-oriented loads on various pages.  Like in a cooking show, I’ve taken all the waiting out, so let’s skip straight to the results!

What we’re looking at here is the performance of the deafult open-source Reddit install under a steadily increasing read load, broken down by layer of the stack:

At first, it performs like a champ. But as the number of concurrent users rises over time, we see that requests slow down. In fact, it looks like we are spending a lot of time per-request in nginx.

We also have access to machine metrics here (blue bar at bottom), so I’ve pulled up the load on the box. Our machine is bored–the max load the machine reaches is only 1.06–but it’s serving slowly! This is a sign that we might not have enough worker threads in our application layer.

In fact, the default Reddit install only sets up a single uwsgi worker. So, let’s fix that, and move on. Here’s what it looks like with 10 uwsgi processes, same workload:

It seems that we’ve traded our uwsgi queuing problems for an overloaded machine, but at least it’s fully utilizing the hardware now–and our throughput is much greater!

Test 2: ramp-up write threads

[global]
run_time: 1800
rampup: 1800
results_ts_interval: 10
progress_bar: on
console_logging: off
xml_report: off

[user_group-1]
threads: 10
script: vote_comment.py

[user_group-2]
threads: 10
script: submit_comment.py

This test will vote and submit comments on a particular thread with inceasing numbers of logged in users.  Ok, go!

One really interesting thing is that we can see there are two distinct trends in the data–one band grows slower faster than the other.  Selecting them for comparison, we can see that the slower band is for rendering the comments, while the faster one is the POST requests for commenting/voting:

We might have expected to see contention for the database (in this case, postgres).  However, by pushing the limits with our load tests, we figured out that the actual limiting factor will be cores on our app servers (or, in this case, server) before we have to worry about the database.  Here’s what the breakdown by layer of stack looked like–note that we’re spending almost no time in our database calls (measured through sqlalchemy and the Cassandra client):

Where to go from here:

• Performance testing is not only valuable to ensure that a new Web app meets projected demand; it can also be part of your CI system to detect performance regressions during everyday development. Here’s a screencast about getting performance tests running in Jenkins.
• If your website is particularly AJAX-heavy, you may also want to do load testing that simulates a browser better and execute JavaScript in order to create the exact load patterns that users will. This makes testing significantly more resource intensive as it requires spinning up headless browsers, but can be accomplished using selenium or a hosted selenium service.
• Tracelytics performance monitoring and analysis isn’t only for load tests; most of our customers run our lightweight instrumentation in production as well as development environments. Sign up for a free trial—no credit card required.

In this two-part series, I’ll walk through generating load using the Python multi-mechanize load testing framework, then collect and analyze data about app performance using Tracelytics.

Also, a request: there’s mechanize documentation available, but I unfortunately haven’t found any full documentation of the python mechanize API online–post a comment if you know where to find it!

Meet the app: Reddit

The web app that I’ll be using for all the examples in this post is an open source reddit running on a single node in EC2. You don’t need to understand how it works in order to enjoy this post, but if you do want to play along, there is a super–easy install script that sets up a whole stack.

Generating the data

Performance testing can start off simple: hit pages in your app, and monitor how long they take to load. You can automate this using something like the mechanize library in Python, or even something more low level like httplib/urllib2.  This is a good start, but today we’re looking for concurrency as well.

Enter multi–mechanize. Multi–mechanize takes simple request or transaction simulation scripts you’ve written and fires them repeatedly from many threads simultaneously in configurable patterns. It’s as easy as writing a few scripts that simulate users doing different actions on your website (login, browse, submit comment, etc.) and then writing a short config that tells multi–mechanize how to play them back.

Since our site is reddit, I’m going to write a few scripts that read discussion threads, one that posts comments, and one that votes on comments. I’ll walk through writing to of the scripts: a simple read–only request, and more complicated one that logs in and submits a comment. The rest of them are available in the full source of the load tests on github.

A simple mechanize script

import mechanize
import time

class Transaction(object):
    def __init__(self):
        self.custom_timers = {}
        self.base_url = 'http://reddit.tracelytics.com'

    def run(self):
        br = mechanize.Browser()
        br.set_handle_robots(False)

        start_timer = time.time()
        resp = br.open(self.base_url + '/r/all/')
        resp.read()
        latency = time.time() - start_timer
        self.custom_timers['All_Items'] = latency
        assert (resp.code == 200), 'Bad HTTP Response'

if __name__ == '__main__':
    trans = Transaction()
    trans.run()
    print trans.custom_timers

First, the test is wrapped in a Transaction class—this is how multi–mechanize will run each of your scripts. The __init__() is called once per worker thread, then run() will be invoked repeatedly to generate the requests of your transaction. For development and debugging, it’s easier to just run the scripts individually, so the __main__ block at the end provides that functionality.

All of the work here is happening in the run() method. A mechanize browser is instantiated, and our simple request for the front page of Reddit is performed. Finally, we make sure that a valid page was returned.

There’s one more thing: custom timers.  Multi–mechanize can collect timing information about the requests it performs. If you store that information in a correctly-named dict, it will be able to generate charts of the data later.

A more complicated script

Now, let’s take a look at a slightly more complicated script. This one posts a comment on a particular story, so it’ll have to take the following actions:

• Log in as a user
• Open thread page on Reddit
• Post comment

It’s a bit longer, so I’ve broken it up according to the bullets above with accompanying explanation.  The full version of the example can be found here:  https://gist.github.com/1529242

import mechanize
import urllib

BASE_URL = 'http://my-reddit'
THREAD = BASE_URL + '/r/reddit/particular-thread'

class Transaction(object):
    def __init__(self):
        self.user = 'redditor'
        self.pass = 'password'
    def run(self):
        # init browser and cookie jar
        br = mechanize.Browser()
        br.set_handle_redirect(True)
        br.set_handle_referer(True)
        br.set_handle_robots(False)
        # do login
        _ = br.open(BASE_URL)
        br.select_form(nr=1)
        br.form['user'] = self.user
        br.form['passwd'] = self.pass
        r = br.submit()
        r.read()
        assert (r.code == 200), 'Bad HTTP Response'

The first thing that’s happening here is familiar: pulling up a page in our mechanize browser. In order to login, though, we need to start interacting with forms on the page, and this means tweaking some of the default mechanize browser settings.  We need to set three attributes on our browser: follow 30x redirects (lets the login redirect back), specify the Referrer page header (validation for comment post), and ignore robots.txt rules (Reddit doesn’t like robots playing human).

After that, it’s on to forms.  The mechanize browser interface with forms is pretty simple: you can list all the forms on the page with browser.forms, select a form to interact with using select_form, and then manipulate the fields of the form using the browser.form object.

select_form  can take a variety of selection predicates, most of which revolve around using attributes such as the form’s CSS ID. Our example, Reddit, doesn’t have much identifying information associated with forms, so I’ve used numeric selection to grab them. The login form happens to be form 1.

        # go to comment page
        posting = THREAD
        rval = THREAD.split('/')[4]
        r = br.open(posting)
        r.read()
        assert (r.code == 200), 'Bad HTTP Response'

Pretty straightforward: head to the thread page now that we’re logged in.  Now we want to actually submit the comment.  Here’s the heavy lifting:

        # load required value from hidden form field
        br.select_form(nr=0)
        uh = br.form['uh']

        # select top-level comment form
        br.select_form(nr=12)
        thing_id = br.form['thing_id']
        id = '#' + br.form.attrs['id']

        # build and submit the comment posting
        data = {'uh':uh, 'thing_id':thing_id, 'id':id, 'renderstyle':'html',
                'r':rval, 'text':"This is such an interesting comment!"}
        nd_dict = dict((k, urllib.quote(v).replace('%20', '+')) for\
                        k,v in data.iteritems())
        new_data = '&'.join("%s=%s" % (k,v) for k,v in nd_dict.iteritems())

        r = br.open(BASE_URL + '/api/comment', new_data)
        r.read()
        assert (r.code == 200), 'Bad HTTP Response'

if __name__ == '__main__':
    trans = Transaction()
    trans.run()

Comment submission is a little bit different because it works via AJAX.  The mechanized browser doesn’t process JavaScript, meaning that we’ll have to take things into our own hands here.  So, we inspect two forms to grab the state information that JavaScript on the page would use to submit the form, and we construct our own request manually.  (Form 0 provides the ‘uh’ value in a hidden field; form 12 is the top-level comment submit form.)

In this simple example, user credentials are provided in __init__. However, a more realistic example might involve many different users logging in. In the code on github, I’ve written a user pool implementation that takes care of this problem by instantiating a pool of logged in users for each script (then, each invocation of run can check out a different user).

(Debugging note for those playing along: if comments are not showing up in the thread but are showing up in the users profile, that means that some of the background jobs may not be running correctly. The site re-caches the comment tree asynchronously after posts.)

Running the full load test

After writing a few individual mechanize scripts, the final step is putting them all together with a multi–mechanize config. Multi-mechanize organizes load tests in terms of “projects” which are represented by subdirectories of a directory called projects. Each project contains a config file and a directory called test_scripts which contains your individual load tests scripts. It should look like this:

The config file specifies how long the load test should run for, whether it should ramp up the amount of pain or keep it constant,  a few output and statistics settings, and of course the number of threads and scripts you want to run. Here’s an example config:

[global]
run_time: 240
rampup: 0
results_ts_interval: 20
progress_bar: on
console_logging: off
xml_report: on

[user_group-1]
threads: 20
script: view_base.py

[user_group-2]
threads: 20
script: view_all.py</pre>

Runtime sets the duration of the test, in seconds. Ramp up, if nonzero, tolls multi-mechanize to linearly increase the number of threads up to the specified numbers during the wrapup.

And here’s how to invoke them, finally:

$ multi-mechanize reddit_read
  user_groups:  2
  threads: 40

[=          6%           ]  14s/240s   transactions: 680  timers: 680  errors: 0

Learning from our load tests

Multi—mechanize collects statistics about timing information that you provide in your tests (custom_timers) and dumps the output in a results subdirectory of your project. This can easily be plotted in your favorite graphics package.  Here’s an example of the average times from a read load increasing over 30 minutes:

Ok, so it’s getting slower, but why??  These timers treat the application like a black box—they’ll show you that it can be slow, but you won’t know why or what layers of the stack are slow. In the next post, we’ll talk about how to gather actionable data from your load tests.

1. Simplified app presentation

Apps are now displayed as an entry point (where requests begin to be traced in the stack) as well as the other component layers and hosts.  Clicking a layer expands the list of hosts, and we automatically remove layers and hosts that aren’t tracing or get moved to other apps.  We’ve focused on layers as the components of your stack are likely to be much more static than the hosts they run on.

2. Seamless instrumentation of new layers and hosts

It’s super-easy to add and configure new apps; now it’s easy to install new instrumentation as well, with built-in install walkthrough.

3. Get notified when new layers and hosts appear

One of the best things about tracing is that it can be extended as you install modules that provide instrumentation for different hosts and parts of your stack.  Now you can keep track of changes in app instrumentation across your deployment.

4. App renaming and deletion

Just in time for spring cleaning: you can now rename and delete apps.  Renamed apps will keep their current data and configuration, but deletion is a one-way street.

5. Search

Trying to make sure everything is in the right place across your handful of deployments?  We’ve got search for layers and hosts both within as well as across all apps.

Ok!  You can check it out live on your app, or sign up to try out tracing.

The Whole Sandwich: Performance by Layer

Structure is simply the call graph of your application. Complex structure is what makes distributed apps hard to debug and even harder to optimize. Each trace has its own structure, and the relationships within that graph can make or break the performance of a web app.

When measuring performance, people typically throw away structural information and keep only timing information.  But in order to answer sophisticated questions about web application performance, it helps to know about both the performance and structure of your app.  To make this concrete, let’s track down a problem in a PHP webapp using Tracelytics.

(click to enlarge)

We’ll start at the app page, looking at the stacked area chart. Remember that this visualization shows us the average request time, broken down by the time spent in each layer. The first thing to notice here is that the performance is wildly inconsistent. Secondly, the variation seems to be largely in PHP and php_mysql, where we have client-side instrumentation of MySQL queries. The largest spikes here put the response time between 200 and 500 ms, but more typically 50 to 100ms. Nothing too surprising here yet — the variance is a bit worrying, but with fewer than 9,000 traces per week we don’t expect to see the most consistent measurements.

Perhaps this variance is due to poorly performing queries–why don’t we take a look at an individual layer of our stack next?

Examining Layer Performance

Let’s take a look at the queries the app is running by drilling down on the php_mysql layer, where we might expect to find some slow queries.  You might do this with the slow query log, but we can get more info out of a heatmap:

(click to enlarge)

What we find is something a little less obvious.  Recall that the heatmap shows us the distribution of calls, broken down by duration, over a period of time–like a bunch of histograms next to each other. From the stacked area chart, we saw a lot of time spent in php_mysql, but this graph shows that nearly every call (up to 98%, the default cutoff) completes faster than 12ms, with the majority returning faster than 2ms. Even more striking, the heatmap is showing a total count of more than 2 million calls — which might  sound like a lot of queries per trace given the 9,000 traces we saw in the stacked area chart.  Where are they all coming from?

Structured Data to the Rescue 

(click to enlarge)

By now, what’s going on may be obvious, but let’s examine the trace view, a visualization of the structure of an individual request, to confirm our intuition.  This particular request actually made over 1600 calls to MySQL, most of which were very similar. While each query examined individually is fast, well-indexed, and nicely behaved, the overall request is inconsistent and slow. This is a classic case of putting a query inside of a loop. If we instead retrieved all the needed data at once, we could iterate over it much faster, saving a lot of overhead in round trips and query execution.

In this case, the structure of the request unnecessarily introduced both an absolute delay and an increase in variability to the page load time. Each layer, examined individually, was performing well, but that’s not enough to guarantee good performance. The structure of the application and the quality of the code play just as much of a role as DB tuning and scaling metrics.

“The key to performance is elegance, not battalions of special cases.”
– Jon Bentley and Douglas McIlroy