为了方便工作,我需要从enwiki-pages-articles.xml中过滤类别,并将它们存储相同的XML格式的类别。因此我选用了SAX解析器,在PyPy和CPython中都适用的包装器解析。对外的原生编译包(同事在PyPy和CPython 中) 。
代码非常简单:
class WikiCategoryHandler(handler.ContentHandler): """Class which detecs category pages and stores them separately """ ignored = set(('contributor', 'comment', 'meta')) def __init__(self, f_out): handler.ContentHandler.__init__(self) self.f_out = f_out self.curr_page = None self.curr_tag = '' self.curr_elem = Element('root', {}) self.root = self.curr_elem self.stack = Stack() self.stack.push(self.curr_elem) self.skip = 0 def startElement(self, name, attrs): if self.skip>0 or name in self.ignored: self.skip += 1 return self.curr_tag = name elem = Element(name, attrs) if name == 'page': elem.ns = -1 self.curr_page = elem else: # we don't want to keep old pages in memory self.curr_elem.append(elem) self.stack.push(elem) self.curr_elem = elem def endElement(self, name): if self.skip>0: self.skip -= 1 return if name == 'page': self.task() self.curr_page = None self.stack.pop() self.curr_elem = self.stack.top() self.curr_tag = self.curr_elem.tag def characters(self, content): if content.isspace(): return if self.skip == 0: self.curr_elem.append(TextElement(content)) if self.curr_tag == 'ns': self.curr_page.ns = int(content) def startDocument(self): self.f_out.write(" \n") def endDocument(self): self.f_out.write(" \n") print("FINISH PROCESSING WIKIPEDIA") def task(self): if self.curr_page.ns == 14: self.f_out.write(self.curr_page.render()) class Element(object): def __init__(self, tag, attrs): self.tag = tag self.attrs = attrs self.childrens = [] self.append = self.childrens.append def __repr__(self): return "Element {}".format(self.tag) def render(self, margin=0): if not self.childrens: return u"{0} ".format( " "*margin, self.tag, "".join([' {}="{}"'.format(k,v) for k,v in {}.iteritems()])) if isinstance(self.childrens[0], TextElement) and len(self.childrens)==1: return u"{0} {3} ".format( " "*margin, self.tag, "".join([u' {}="{}"'.format(k,v) for k,v in {}.iteritems()]), self.childrens[0].render()) return u"{0} \n{3}\n{0} ".format( " "*margin, self.tag, "".join([u' {}="{}"'.format(k,v) for k,v in {}.iteritems()]), "\n".join((c.render(margin+2) for c in self.childrens))) class TextElement(object): def __init__(self, content): self.content = content def __repr__(self): return "TextElement" def render(self, margin=0): return self.content
这个任务使用MongoDB作为数据来源,并对结构进行重新分配。算法是:
for each category.id in redis_categories (it holds *category.id -> category title mapping*) do: title = redis_categories.get(category.id) parent_categories = mongodb get categories for title for each parent_cat in parent categories do: redis_tree.sadd(parent_cat, title) # add to parent_cat set title
结论
进行的测试仅仅是我最终工作的一个简介。它需要一个知识体系,一个我从抽取维基百科中适当的内容中得到的知识体系。
PyPy相比CPython,在我这个简单的数据库操作中,提高了2-3倍的性能。(我这里没有算上SQL解析器,大约8倍)
多亏了PyPy,我的工作更加愉悦了——我没有改写算法就使Python有了效率,而且PyPy没有像CPython一样把我的CPU弄挂了,以至于一段时间内我没法正常的使用我的笔记本了(看看CPU时间占的百分比吧)。
任务几乎都是数据库操作,而CPython有一些加速的乱七八糟的C语言模块。PyPy不使用这些,但结果却更快!
我的全部工作需要大量的周期,所以我真高兴能用PyPy。
查看更多关于PyPy和CPython的性能比较测试的详细内容...
声明:本文来自网络,不代表【好得很程序员自学网】立场,转载请注明出处:http://www.haodehen.cn/did86845