Profiling is a technique used to measure the performance of a program, specifically in terms of the amount of time and memory it consumes. In Python, there are several tools and libraries available for profiling, which can help you identify bottlenecks in your code and optimize it for better performance.
In this article, we will focus on time profiling. We will see why it is important, what tools you can use in Python to assist you in this task, and a practical example of where profiling can help you optimize a portion of code.
Why use profiling?
Profiling is an essential step in the software development process, as it helps you identify and fix performance issues early on. This can save you a lot of time and effort in the long run, as it allows you to avoid costly and time-consuming rewrites later on.
Additionally, profiling can help you identify potential scalability issues with your code. As your codebase grows and your application handles more traffic, or your algorithm more data, it's critical to make sure that it can continue to perform efficiently. Profiling can help you identify areas of your code that may not scale well, so you can optimize them before they become a problem.
Finally, each language has its specificities when it comes to performance, and certain built-in objects or functions can be much more efficient than others. In these situations, profiling can help detect the precise moment that is taking longer and can result in significant cost savings over time, as well as improved user experience.
How to profile Python code
There are several approaches to profiling Python code, each with its own advantages and disadvantages. It is important to note that each method has a specific purpose and can be used in conjunction with one another. Here are some of the most commonly used libraries:
timeit module
The timeit module is probably the most basic and popular tool on this list. It is a built-in Python library that allows you to measure the execution time of small bits of code. It's useful for comparing the performance of different algorithms or code snippets.
To use timeit, you first need to import it and then call the timeit.timeit() function, passing in the method you want to profile. The function will execute the code multiple times and return the average execution time.
The timeit module is advantageous for comparing built-in functions or short code segments due to its reliability, which comes from repeated execution. However, it does not provide insight into where time is being spent within the code, making it unsuitable for longer methods.
cProfile module
The cProfile module is a built-in Python library that provides more detailed profiling information than timeit. You can use it to measure the execution time of each function in your code, as well as the number of times each function is called.
To use cProfile, you first need to import it and then run the cProfile.run() function, passing in the code you want to profile as a string. The function will execute the code and print a summary containing detailed profiling information about the execution – we will see what those results look like later on.
You can also save the results to a file, and read them later with the pstats module, which also allows you to sort the results as you wish.
The output will show you a list of all the methods called during the execution, including built-ins. For each one, it shows the number of times it was called and the time that was spent inside the method. Two different measures are proposed: tottime, which excludes the time spent in inner functions, and cumtime which includes this time.
Third-party profilers
Other third-party profilers are available for Python to assist with more specific operations. For instance, line_profiler can show which line is consuming the most time, while memory_profiler (no longer actively maintained) can be used to identify which part of the code is utilizing the most memory. Additionally, snakeviz provides a visual representation of the profiling results.
Use cProfile to optimize your code
After this overview of the different tools that can be used to profile your code, let’s see how we can use cProfile to optimize a code and make it run faster. Let’s consider the following function, which takes in two lists of names, and remove the names in the second list from the first one. Since we don’t know whether the names are capitalized or not, we use the lower built-in function to compare them to know if they should be removed, in the should_remove method.
However, this function is pretty slow as it takes more than 4 seconds for a list of 18000 names with 2000 to remove. Let’s run it through cProfile to see how we could optimize it. We get the following output :
First, note the execution is slower since cProfile is adding some processing to measure everything. In this case, the execution time climbs from 4 seconds to 11 seconds. This is one of the inconveniences of this tool: the overhead is quite high when compared to other solutions, and it is highly correlated with the number of function calls. We can however notice that a lot of time is spent in the built-in lower method, which is called 68,958,000 times, for a total of almost 4 seconds.
Indeed, we apply the lower function only when comparing two names together from the two lists, which means we apply the lower function to the same names multiple times, resulting in a huge waste of processing power. Instead, let’s apply the lower function to all the names at the beginning, and then compare the names together.
When re-running the function, the execution now takes less than a second, instead of more than 4 seconds previously! Using cProfile, we get the following result:
The reduction in execution time with cProfile is even more drastic, from 11s to 1s. This is because we reduced the number of function calls, hence reducing the overhead induced by cProfile. Indeed, lower is now only called 20,239 times, which corresponds to the number of names in the two lists. It is therefore important to keep in mind that if cProfile can help us identify slow parts of our code, it should not be used to compare execution times directly, as these can vary depending on the implementation. To solve this problem, other libraries such as pyInstrument take a different approach to avoid penalizing implementations with many function calls.
Obviously, this example was pretty basic, and we probably could have optimized it without using a profiler. But as soon as the code becomes more complex and the number of methods begins to rise, a profiler can help you save a lot of time, both in development and execution time.
If you are interested in learning more about profiling and seeing a different point of view with a focus on more visual representations, make sure to check this other article on our blog.
And if you ever need help with a data science or data engineering project, feel free to contact us!
%20(1).png){kind=logo}