How to Profile Python Code With cProfile

Programming is a sequential job involving writing code, testing, monitoring (profiling), reviewing the reports and results, and refactoring. After all, there’s no guarantee that the first written source code is in its best-optimized shape.

In this article, we're going to focus on the third step of the sequential procedure, which is profiling, with respect to cProfile in Python.

Profiling explains the techniques for determining the parts and units of a program that take a tremendous amount of time to execute. With it, we can use the reports and numbers to improve performance as well as the execution time of functions and methods.

It’s important to note that profiling is not always the best answer for better performance. However, it’s suitable when we need some extra speed through our executions or if there is a huge request load over the servers and we need to manage the complexity of the components.

If you’re already satisfied with your program's execution time, there’s no need to bother with cProfile. However, it can be used when execution time has become a concern and it breaks other parts of the service.

For instance, your server is experiencing a lot of requests and heavy traffic owing to advertising efforts and users have noticed that it is responding slowly. This is where cProfile will come in handy if the service is Python-based and some Python function calls are causing the heavy traffic load.

Profiling is a game-changer when it comes to redesigning and restructuring a project and the codebase. Most times, you don't need to put effort into a full redesign task as a simple R&D and profiling would work.

In Python and *nix environments, there are many packages and Python libraries that provide facilities for evaluating the cost of source codes, whether it's security threats or the time it takes for a component to execute.

cProfile in Python is categorized as a standard library. In the next section, we'll use an example followed by the actual implementation and possible strategies for optimization. It’s a good illustration of what happens in real-world cases.

Scenario: There’s a large text file of novel titles separated by new lines. Some names have occurred twice or even multiple times. The goal is to read the file and create a local variable filled with all the novel names without duplicate names included. The main requirement is to use multiple functions as the solution.

Here is a short scheme of what we have in the text file.

Solution: We need functions for reading the file, ignoring the duplicated values, and a main function for calling them in order.

• read_file(path: str) -> list: Gets the file name/path and returns the list of titles.

• ignore_duplicates(titles: list): Gets a list of titles (the value returned from read_file()) and returns a duplicate-safe list.

• main(): Main function.

Keep in mind that the following code snippet is not perfect. We will come up with solutions with the help of cProfile to optimize the code and improve the execution time of each function.

Executing the above script will show the safe_titles variable that contains the unique titles. On a lower-end machine, the same script takes more seconds to run and that's why we're not done yet.

Let's begin the cProfile part by importing this module first.

At the very bottom of the script, replace the phrase main() with cProfile.run('main()'). It not only deals with its parameter as an interpreting line, but illustrates the information we need for optimizing.

In the first line, we have 321630 function calls in 0.5 seconds. This execution time is almost 5 seconds on the low-end machine.

We have two ~160K calls in two lines. To sort the depicted table, simply add the sort=COLUMN attribute to the cProfile.run() method.

Here is the result.

We can see which lines get executed the most. We have two lines with over +160K executions! That's massive. It's time to resolve things.

read_file() optimization: We're looping through all names in order to remove the \n suffix and append the name into another list that is described like {method 'append' of 'list' objects} in the first row.

That's a heavy load on the system. Instead, we could do it slightly cleaner by returning file.read().splitlines() right within the with scope as follows. It's an impressive O(n) to O(1) complexity.

ignore_duplicates() optimization: Next, it's time to refactor the core function. There is an O(1) trick that we can use to make a list filled with unique items. This involves using sets.

Now, our source code should look like the following snippet after all optimizations and refactors are applied.

Let’s execute the program and wait for the result of cProfile to shine through even faster than before!

There is a huge performance improvement here. We went from >300K function calls to only 15 calls. We also made it much faster, from 0.5s to 0.01, which is ~x50 quicker than before!

With the applied performance improvements, we can have the same functionality but 50 times faster on the low-spec machine as well.

Another way of applying cProfile through components is via decorators. Simply structure cProfile objects and with the help of io and pstats modules, we can make our cProfile much easier to use and more precise. A good matching module that helps manipulate the statistics returned from cProfile is pstats.

In the next section, we'll share some tricks to turn the solid raw information that cProfile provides us into better graphical shapes and reports using a third-party library in Python.

There are online services and tools that can be used to illustrate the output and reports of a cProfile Python execution. It's a nice practice for those who need the reports for meetings and technical events.

A quick library installation is needed. We assume that you've already installed virtualenv and have your environment activated.

Now, we need to export the output of cProfile into a .prof file. The following commands do so and keep the statistics in a file named program.prof.

As a last step, run the snakeviz command and it will illustrate the reports right on your browser.

Browse the link and you'll see a panel like this.

You can customize and filter the shown data as you wish. Feel free to work with the utilities that the tool provides.

Redesigning a codebase is costly but with the help of monitoring and profiling methodologies, you can avoid full redesigns and optimize it instead. In the cProfile example discussed in this article, you'll have plenty of information for optimizing as well as an accurate report of how your functions work over time. Your team can have sessions to analyze the results obtained from libraries and modules, such as cProfile in Python and pstat, for better performance.