Accelerating Data Processing with Pandas and Python: The Need for Speed
=================================================================================================
In the world of programming, Python's performance is often a topic of discussion. This is due, in part, to its dynamic nature and abstraction costs, which can impact execution speed compared to lower-level languages. However, a recent study conducted by an expert programmer aims to shed light on the misconceptions surrounding Python's performance.
The study focused on a large, publicly available dataset, the 7+ Million Company dataset, totaling approximately 24.5 GB of data and containing 185 million rows. The workload involved a list of 24 file paths, with each file having an average size of around 1 GB.
The author of the study, seeking answers to these performance-related questions, implemented a code named "parallelism.py" on their GitHub account. The code was designed to handle the workload using both single-threaded and multi-processing approaches.
The experiment was carried out on a system with a power supply of 600 watts. Interestingly, during the runtime of the multi-processing approach, there was a doubling of power consumption compared to the single-threaded approach. However, the multi-processing approach used 31.77% less power than the single CPU approach during the entire experiment, demonstrating its energy efficiency.
Modern CPUs are multi-core and offer multiple rail lines for workloads. In this context, Python can be compared to train tracks, where using multiple rail lines can be quicker. The multi-processing approach, using the Pool() class, chunked up the work and spread it to available workers.
In the single-threaded approach, the workload took 411.92 seconds to complete. In contrast, the multi-processing approach completed the same workload in just 140.03 seconds, a 3.02 times faster execution time. The final dataframe produced by the multi-processing approach had 192 rows and 6 columns.
The author's findings support the notion that Python's reputation for slowness is largely a myth. Proper use of profiling, writing efficient code, leveraging JIT-enabled interpreters, and using compiled extensions can greatly improve Python program performance.
The study also highlights the ongoing debate about whether Python is slow or if it's the coding methods that slow it down. The author, recognising the importance of addressing this misunderstanding, has written an article about speeding up Python and Pandas using Rapids and CuDF.
The 7+ Million Company dataset used in the experiment is licensed under the creative commons CC0.01 and can be requested from People Data Labs. The author encourages other developers to conduct similar experiments and share their findings to continue the conversation about improving Python performance.
In addition, the HPI has launched the clean IT initiative to develop climate-friendly digital solutions and AI applications through first-class training offers and the sustainable and energy-efficient use of IT systems. This initiative aligns with the author's findings, emphasising the importance of energy-efficient coding practices in the digital age.
- The study on Python's performance, conducted using the 7+ Million Company dataset, showcases the effectiveness of data-and-cloud-computing techniques in optimizing the execution of large workloads through the use of modern technology like multi-processing.
- Efficient use of technology, such as data-and-cloud-computing and multi-processing approaches, plays a crucial role in enhancing Python's performance and promoting energy efficiency, as demonstrated in the recent study on Python's performance.
