破坏巨大的xml文件后插入数据库的有效方法

问题描述

我需要拆分一个巨大的xml文件，然后将其插入数据库，这是做我正在做的事的最有效方法

这是我的代码

import xml.etree.cElementTree as etree
filename = r'D:\test\Books.xml'
context = iter(etree.iterparse(filename, events=('start', 'end')))
_, root = next(context)
books = []
for event, elem in context:
    if event == 'start' and elem.tag == '{http://www.book.org/Book-19200/biblography}Book':
        etree.register_namespace("", "http://www.book.org/Book-19200/biblography")
        xml = etree.tostring(elem)
        xmls.append(xml)
        if len(xmls) == 100:
            populate_db(books)
            books = []
        root.clear()

def populate_db(books):
    c.executemany('INSERT INTO Books VALUES (?)', books)

我的样本book.xml看起来像这样

<Books>
    <Book xmlns="http://www.book.org/Book-19200/biblography"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    ISBN="519292296"
    xsi:schemaLocation="http://www.book.org/Book-19200/biblography ../../book.xsd 
    http://www.w3.org/2000/12/xmldsig# ../../xmldsig-core-schema.xsd">
        <Detail ID="67">
            <BookName>Code Complete 2</BookName>
            <Author>Steve McConnell</Author>
            <Pages>960</Pages>
            <ISBN>0735619670</ISBN>        
            <BookName>Application Architecture Guide 2</BookName>
            <Author>Microsoft Team</Author>
            <Pages>496</Pages>
            <ISBN>073562710X</ISBN>
        </Detail>
    </Book>
    <Book xmlns="http://www.book.org/Book-19200/biblography"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    ISBN="519292296"
    xsi:schemaLocation="http://www.book.org/Book-19200/biblography ../../book.xsd 
    http://www.w3.org/2000/12/xmldsig# ../../xmldsig-core-schema.xsd">
        <Detail ID="87">
            <BookName>Rocking Python</BookName>
            <Author>Guido Rossum</Author>
            <Pages>960</Pages>
            <ISBN>0735619690</ISBN>
            <BookName>Python Rocks</BookName>
            <Author>Microsoft Team</Author>
            <Pages>496</Pages>
            <ISBN>073562710X</ISBN>
        </Detail>
    </Book>
</Books>

这种方法是追加到List并使用100个批次并有效地插入数据库还是在这里需要考虑多线程