当前位置: 代码迷 >> 综合 >> Python Pickle报:OverflowError: cannot serialize a bytes object larger than 4 GiB的解决方法
  详细解决方案

Python Pickle报:OverflowError: cannot serialize a bytes object larger than 4 GiB的解决方法

热度:45   发布时间:2023-12-16 06:47:41.0

按照这里的经验

直接在pickle.dump中增加一个protocol = 4这个参数就行:

import picklepickle.dump(data,open('file','wb'),protocol = 4)

不过,具体为什么要这样做呢?可以参考一下这里:

https://docs.python.org/3/library/pickle.html#data-stream-format

  • Protocol version 0 is the original “human-readable” protocol and is backwards compatible with earlier versions of Python.

  • Protocol version 1 is an old binary format which is also compatible with earlier versions of Python.

  • Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes. Refer to PEP 307 for information about improvements brought by protocol 2.

  • Protocol version 3 was added in Python 3.0. It has explicit support for bytes objects and cannot be unpickled by Python 2.x. This was the default protocol in Python 3.0–3.7.

  • Protocol version 4 was added in Python 3.4. It adds support for very large objects, pickling more kinds of objects, and some data format optimizations. It is the default protocol starting with Python 3.8. Refer to PEP 3154 for information about improvements brought by protocol 4.

  • Protocol version 5 was added in Python 3.8. It adds support for out-of-band data and speedup for in-band data. Refer to PEP 574 for information about improvements brought by protocol 5.

总共有6种不同的protocol,其中4是从Python 3.4开始支持的,主要增加了对超大对象操作的功能。简单总结一下。

  相关解决方案