A MemoryError means that the interpreter has run out of memory to allocate to your Python program. This may be due to an issue in the setup of the Python environment or it may be a concern with the code itself loading too much data at the same time.
An Example of MemoryError
To have a look at this error in action, let’s start with a particularly greedy piece of code. In the code below, we start with an empty array and use nested arrays to add strings to it. In this case, we use three levels of nested arrays, each with a thousand iterations. This means at the end of the program, the array s has 1,000,000,000 copies of the string "More."
s = []
for i in range(1000):
for j in range(1000):
for k in range(1000):
s.append("More")Output
As you might expect, these million strings are a bit much for, let’s say, a laptop to handle. The following error is printed out:
C:\code\Python\MemErr\venv\3K\Scripts\python.exe C:/code/python/MemErr/main.py
Traceback (most recent call last):
File "C:/code/python/MemErr/main.py", line 6, in <module>
s.append("More")
MemoryErrorIn this case, the traceback is relatively simple as there are no libraries involved in this short program. After the traceback showing the exact function call which caused the issue, we see the simple but direct MemoryError.
Two Ways to Handle A MemoryError in Python
Appropriate Python Set-up
This simplest but possibly least intuitive solution to a MemoryError actually has to do with a potential issue with your Python setup. In the event that you have installed the 32-bit version of Python on a 64-bit system, you will have extremely limited access to the system's memory. This restricted access may cause MemoryErrors on programs that your computer would normally be able to handle.
Attention to Large Nested Loops
If your installation of Python is correct and these issues still persist, it may be time to revisit your code. Unfortunately, there is no cut and dry way to entirely remove this error outside of evaluating and optimizing your code. Like in the example above, pay special attention to any large or nested loops, along with any time you are loading large datasets into your program in one fell swoop.
In these cases, the best practice is often to break the work into batches, allowing the memory to be freed in between calls. As an example, in the code below, we have broken out earlier nested loops into 3 separate loops, each running for 333,333,333 iterations. This program still goes through one million iterations but, as the memory can be cleared through the process using a garbage collection library, it no longer causes a MemoryError.
An Example of Batching Nested Loops
import gc
s = []
t = []
u = []
for i in range(333333333):
s.append("More")
gc.collect()
for j in range(333333333):
t.append("More")
gc.collect()
for k in range(333333334):
u.append("More")
gc.collect()
How to Avoid a MemoryError in Python
Python’s garbage collection makes it so that you should never encounter issues in which your RAM is full. As such, MemoryErrors are often indicators of a deeper issue with your code base. If this is happening, it may be an indication that more code optimization or batch processing techniques are required. Thankfully, these steps will often show immediate results and, in addition to avoiding this error, will also vastly shorten the programs' runtime and resource requirements.
Track, Analyze and Manage Errors With Rollbar
Managing errors and exceptions in your code is challenging. It can make deploying production code an unnerving experience. Being able to track, analyze, and manage errors in real-time can help you proceed with more confidence. Rollbar automates error monitoring and triaging, making fixing Python errors easier than ever. Sign Up Today!
