python读取一个大于10G的txt文件的方法

第python读取一个大于10G的txt文件的方法用python读取一个大于10G的文件，自己电脑只有8G内存，一运行就报内存溢出：MemoryError

python如何用open函数读取大文件呢？

读取大文件

首先可以自己先制作一个大于10G的txt文件

a='''

2025-02-0221:33:31,678[django.request:93][base:get_response][WARNING]-NotFound:/44/

2025-02-0221:33:31,679[django.server:124][basehttp:log_message][WARNING]-"HEAD44/HTTP/1.1"4041678

2025-02-0222:14:04,121[django.server:124][basehttp:log_message][INFO]-code400,messageBadrequestversion('HTTP')

2025-02-0222:14:04,122[django.server:124][basehttp:log_message][WARNING]-"GET../../mnt/custom/ProductDefinitionHTTP"400-

2025-02-0222:16:21,052[django.server:124][basehttp:log_message][INFO]-"GET/api/loginHTTP/1.1"3010

2025-02-0222:16:21,123[django.server:124][basehttp:log_message][INFO]-"GET/api/login/HTTP/1.1"2003876

2025-02-0222:16:21,192[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/main_bg.pngHTTP/1.1"2002801

2025-02-0222:16:21,196[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/style.cssHTTP/1.1"2001638

2025-02-0222:16:21,229[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/bg.jpgHTTP/1.1"200135990

2025-02-0222:16:21,307[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/fonts/icomoon.ttfu4m6fyHTTP/1.1"2006900

2025-02-0222:16:23,525[django.server:124][basehttp:log_message][INFO]-"POST/api/login/HTTP/1.1"3020

2025-02-0222:16:23,618[django.server:124][basehttp:log_message][INFO]-"GET/api/index/HTTP/1.1"20018447

2025-02-0222:16:23,709[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/js/commons.jsHTTP/1.1"20013209

2025-02-0222:16:23,712[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/css/admin.cssHTTP/1.1"20019660

2025-02-0222:16:23,712[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/css/common.cssHTTP/1.1"2001004

2025-02-0222:16:23,714[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/js/app.jsHTTP/1.1"20020844

2025-02-0222:16:26,509[django.server:124][basehttp:log_message][INFO]-"GET/api/report_list/1/HTTP/1.1"20014649

2025-02-0222:16:51,496[django.server:124][basehttp:log_message][INFO]-"GET/api/test_list/1/HTTP/1.1"20024874

2025-02-0222:16:51,721[django.server:124][basehttp:log_message][INFO]-"POST/api/add_case/HTTP/1.1"2000

2025-02-0222:16:59,707[django.server:124][basehttp:log_message][INFO]-"GET/api/test_list/1/HTTP/1.1"20024874

2025-02-0322:16:59,909[django.server:124][basehttp:log_message][INFO]-"POST/api/add_case/HTTP/1.1"2000

2025-02-0322:17:01,306[django.server:124][basehttp:log_message][INFO]-"GET/api/edit_case/1/HTTP/1.1"20036504

2025-02-0322:17:06,265[django.server:124][basehttp:log_message][INFO]-"GET/api/add_project/HTTP/1.1"20017737

2025-02-0322:17:07,825[django.server:124][basehttp:log_message][INFO]-"GET/api/project_list/1/HTTP/1.1"20029789

2025-02-0322:17:13,116[django.server:124][basehttp:log_message][INFO]-"GET/api/add_config/HTTP/1.1"20024816

2025-02-0322:17:19,671[django.server:124][basehttp:log_message][INFO]-"GET/api/config_list/1/HTTP/1.1"20019532

whileTrue:

withopen("xxx.log","a",encoding="utf-8")asfp:

fp.write(a)

循环写入到xxx.log文件，运行3-5分钟，pycharm打开查看文件大小大于10G

于是我用open函数直接读取

f=open("xxx.log",'r')

print(f.read())

f.close()

抛出内存溢出异常:MemoryError

Traceback(mostrecentcalllast):

File"D:/2025kecheng06/demo/txt.py",line35,inmodule

print(f.read())

MemoryError

运行的时候可以看下自己电脑的内存已经占了100%，cpu高达91%，不挂掉才怪了！

这种错误的原因在于，read()方法执行操作是一次性的都读入内存中，显然文件大于内存就会报错。

read()的几种方法

1.read()方法可以带参数n，n是每次读取的大小长度，也就是可以每次读一部分，这样就不会导致内存溢出

f=open("xxx.log",'r')

print(f.read(2048))

f.close()

运行结果

2025-10-2421:33:31,678[django.request:93][base:get_response][WARNING]-NotFound:/44/

2025-10-2421:33:31,679[django.server:124][basehttp:log_message][WARNING]-"HEAD44/HTTP/1.1"4041678

2025-10-2422:14:04,121[django.server:124][basehttp:log_message][INFO]-code400,messageBadrequestversion('HTTP')

2025-10-2422:14:04,122[django.server:124][basehttp:log_message][WARNING]-"GET../../mnt/custom/ProductDefinitionHTTP"400-

2025-10-2422:16:21,052[django.server:124][basehttp:log_message][INFO]-"GET/api/loginHTTP/1.1"3010

2025-10-2422:16:21,123[django.server:124][basehttp:log_message][INFO]-"GET/api/login/HTTP/1.1"2003876

2025-10-2422:16:21,192[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/main_bg.pngHTTP/1.1"2002801

2025-10-2422:16:21,196[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/style.cssHTTP/1.1"2001638

2025-10-2422:16:21,229[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/bg.jpgHTTP/1.1"200135990

2025-10-2422:16:21,307[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/fonts/icomoon.ttfu4m6fyHTTP/1.1"2006900

2025-10-2422:16:23,525[django.server:124][basehttp:log_message][INFO]-"POST/api/login/HTTP/1.1"3020

2025-10-2422:16:23,618[django.server:124][basehttp:log_message][INFO]-"GET/api/index/HTTP/1.1"20018447

2025-10-2422:16:23,709[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/js/commons.jsHTTP/1.1"20013209

2025-10-2422:16:23,712[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/css/admin.cssHTTP/1.1"20019660

2025-10-2422:16:23,712[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/css/common.cssHTTP/1.1"2001004

2025-10-2422:16:23,714[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/js/app.jsHTTP/1.1"20020844

2025-10-2422:16:26,509[django.server:124][basehttp:log_message][I

这样就只读取了2048个字符，全部读取的话，循环读就行

f=open("xxx.log",'r')

whileTrue:

block=f.read(2048)

print(block)

ifnotblock:

break

f.close()

2.readline()：每次读取一行，这个方法也不会报错

f=open("xxx.log",'r')

whileTrue:

line=f.readline()

print(line,end="")

ifnotline:

break

f.close()

3.readlines()：读取全部的行，生成一个list，通过list来对文件进行处理，显然这种方式依然会造成：MemoyError

真正Pythonic的方法

真正Pythonci的方法，使用with结构打开文件，fp是一个可迭代对象，可以用for遍历读取每行的文件内容

withopen("xxx.log",'r')asfp:

forlineinfp:

print(line,end="")

yield生成器读取大文件

前面一篇讲yield生成器的时候提到读取大文件，函数返回一个可迭代对象，用next()方法读取文件内容

defread_file(fpath):

BLOCK_SIZE=1024

withopen(fpath,'rb')asf:

whileTrue:

block=f.read(BLOCK_SIZE)

ifblock:

yieldblock

else:

return

if__name__=='__main__':

a=read_file("xxx.log")

print(a)#generatorobjec

print(next(a))#bytes类型

print(next(a).decode("utf-8"))#str

运行结果

generatorobjectread_fileat0x00000226B3005258

b'\r\n2025-10-2421:33:31,678[django.request:93][base:get_response][WARNING]-NotFound:/44/\r\n2025-10-2421:33:31,679[django.server:124][basehttp:log_message][WARNING]-"HEAD44/HTTP/1.1"4041678\r\n2025-10-2422:14:04,121[django.server:124][basehttp:log_message][INFO]-code400,messageBadrequestversion(\'HTTP')\r\n2025-10-2422:14:04,122[django.server:124][basehttp:log_message][WARNING]-"GET../../mnt/custom/ProductDefinitionHTTP"400-\r\n2025-10-2422:16:21,052[django.server:124][basehttp:log_message][INFO]-"GET/api/loginHTTP/1.1"3010\r\n2025-10-2422:16:21,123[django.server:124][basehttp:log_message][INFO]-"GET/api/login/HTTP/1.1"2003876\r\n2025-10-2422:16:21,192[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/main_bg.pngHTTP/1.1"2002801\r\n2025-10-2422:16:21,196[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/style.cssHTTP/1.1"2001638\r\n2025-10-2422:16:21,229[django.server:124]'

[basehttp:log_message][INFO]-"GET/static/assets/img/bg.jpgHTTP/1.1"200135990

2025-10-2422:16:21,307[django.server:124][basehttp:log_messag