首页 > 小工具 > 写了个整理AAAI论文摘要的脚本

写了个整理AAAI论文摘要的脚本

2011年12月1日 发表评论 阅读评论

这几天需要扫AAAI的摘要及论文,但是一看AAAI每年有几百篇,我就傻眼了,加之本人有个习惯,善其事之前,总是喜欢利其器,于是花了个把小时写了个把AAAI的摘要爬出来并整理成一个html文件的程序。

#!/usr/bin/env python
# -*- coding: UTF-8 -*-
#
# Copyright 2011 Sigma  http://www.sigma.me.
#
#
#######usage#######
# $python
# >>>from get_abstract_aaai import *
# >>> get_allyear_abstract(<DBLP_conf_link>)
# or
# >>> get_allyear_pdf(<DBLP_conf_link>)
# DBLP_conf_link is like "http://www.informatik.uni-trier.de/~ley/db/conf/aaai/"
#  OR just modify the main function and then
# $python get_abstract_aaai.py
import os
import urllib
import urllib2
import httplib #for https link
import re
import socket
import locale,string
import codecs

title_regx=r'<div id="title">([\s\S]+)</h3></div>'
author_regx=r'<div id="author"><em>([\s\S]+)</em></div>'
abstract_regx=r'<h4>Abstract</h4>\n\t<br />\n\t<div>([\s\S]*)</div>\n\t<br />\n\t</div>'
article_link_regx=r'</b><br><a href="http://www.aaai.org/ocs/index.php/AAAI/AAAI([a-zA-Z0-9-\s\.\/]+)">'
dblp_oneyear_regx=r'http://www.informatik.uni-trier.de/~ley/db/conf/aaai/aaai([\s\S]*).html'
year_link_regx=r'<a href="([a-zA-Z0-9-\s\.\/]+)">Contents</a>'
#pdf_link_regx=r'href="([a-zA-Z0-9-\s\.\/]+)" class="action" target="_parent">PDF</a>'
pdf_link_regx=r'Full Text:\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t<a href="([\s\S]+)" class="action" target="_parent">PDF</a>'

#get_allyear_abstract("http://www.informatik.uni-trier.de/~ley/db/conf/aaai/")

def get_allyear_abstract(url):
    page=urllib.urlopen(url).read()
    year_links=re.findall(year_link_regx,page)
    for link in year_links:
        expend_link=url+link
        print expend_link
        get_oneyear_abstract(expend_link)



def get_oneyear_abstract(url):
    year=re.findall(dblp_oneyear_regx,url)[0]
    print year
    page=urllib.urlopen(url).read()
    links=re.findall(article_link_regx,page)
    articles=[]
    if links is not None:
        for link in links:
            url='http://www.aaai.org/ocs/index.php/AAAI/AAAI'+link.replace('view','viewPaper')
            article=get_aaai_abstract_from_url(url)
            if article is not None:
                articles.append(article)
        article_write2file(articles,year)
    return

def get_aaai_abstract_from_url(url):
    page=urllib.urlopen(url).read()
    title_val=extractAttrValue_mask(title_regx,page)
#    print title_val
    author_val=extractAttrValue_mask(author_regx,page)
#    print author_val
    abstract_val=extractAttrValue_mask(abstract_regx,page)
#    print abstract_val
    ret_val={'title':title_val.decode('UTF-8'),'author':author_val.decode('UTF-8'),'abstract':abstract_val.decode('UTF-8'),}
    print ret_val
    return ret_val

def article_write2file(articles,year):
    html_file_name='aaai'+year+'.html'
    html_head=u'<HTML>\n<HEAD>\n<TITLE>AAAI '+year+u' Publication & Abstract List</TITLE>\n<META Http-equiv="Content-Type" Content="text/html; charset=utf-8">\n</HEAD>\n<BODY>\n<H1><FONT COLOR="#365f91">AAAI '+year+u' Publication &amp;Abstract List</FONT></H1>\n'
    html_foot=u'</BODY>\n</HTML>'
    html_file=codecs.open(html_file_name,'w','utf-8')
    html_file.write(html_head)
    for article in articles:
        html_file.write(u'<H3><FONT COLOR="#4f81bd">'+article['title']+u'</FONT></H3>\n')
        html_file.write(article['author']+u'\n')
        html_file.write(u'<P><strong>Abstract:</strong>'+article['abstract']+u'</P>\n')
    html_file.write(html_foot)
    html_file.close()

def extractAttrValue_mask(regx,allstring):
    infos = re.findall(regx,allstring)
    attrValue = 'no infomation in AAAI database'
    if(infos):
		    attrValue = infos[0]
    return attrValue

if __name__ == "__main__":
    print "main"
#    get_allyear_abstract('http://www.informatik.uni-trier.de/~ley/db/conf/aaai/')

实际使用中发现,貌似08年以前的论文摘要在AAAI网站上没有,并且很多论文直接用google scholar也搜不到,不过我想,只要对python熟悉些,根据这段代码,还是很容易改革能用的(前提是abstract以及pdf链接不是通过js获得的,果如此,得用一些高级的python库)。



本文作者: Sigma    在新浪微博关注SigmaSigmaWeibo    RSS订阅本博客
本文链接: http://www.sigma.me/2011/12/01/write-col-abstract-and-downlaod-python-file-for-aaa.html
本博客采用知识共享署名—非商业性-禁止演绎使用3.0协议进行许可,转载请保留作者和原文链接。

分类: 小工具 标签: , , ,

  1. syoummer
    2011年12月3日19:14 | #1

    批量下载论文是不符合数据库的版权规定的。。。。。。。

  2. 2011年12月3日19:16 | #2

    @syoummer
    好吧,速度把下论文部分去掉

  3. 2011年12月21日00:11 | #3

    继续学习 收藏

  4. 2019年1月14日23:46 | #4

    My wife and i felt now thankful Emmanuel managed to carry out his inquiry through the entire precious recommendations he made out of your web page. It’s not at all simplistic just to always be freely giving secrets and techniques which the rest may have been selling. And we also grasp we now have the blog owner to thank for that. The most important explanations you have made, the simple website navigation, the relationships you aid to foster – it’s all impressive, and it is assisting our son and the family reckon that that issue is thrilling, which is highly pressing. Thank you for all the pieces!

  5. 2019年1月15日07:47 | #5

    I would like to convey my affection for your generosity supporting those who need help on the question. Your very own dedication to passing the message up and down came to be incredibly advantageous and have in most cases permitted people just like me to get to their desired goals. Your warm and friendly recommendations means so much to me and even more to my colleagues. With thanks; from everyone of us.

  6. 2019年1月15日16:25 | #6

    My husband and i have been delighted when Chris managed to conclude his investigations by way of the ideas he acquired when using the site. It’s not at all simplistic to just continually be offering methods many people may have been making money from. And we fully understand we’ve got the blog owner to appreciate because of that. The illustrations you’ve made, the simple website navigation, the relationships your site help engender – it is most excellent, and it’s aiding our son and the family understand the concept is thrilling, and that’s seriously pressing. Thanks for the whole thing!

  7. 2019年1月16日01:18 | #7

    nike dallas cowboys 92 josh brent limited navy blue throwback alternate jersey
    [url=http://www.nhadatanhtuan.com/womens-nike-air-max-lunar-1-white-yellow-shoesa]womens nike air max lunar 1 white yellow[/url]

  8. 2019年1月16日02:16 | #8

    I simply wanted to thank you very much again. I am not sure what I would have undertaken in the absence of the type of techniques documented by you directly on such a area of interest. Previously it was the difficult circumstance in my circumstances, nevertheless looking at the very professional manner you processed that made me to cry with contentment. I’m just happy for your help and as well , believe you are aware of an amazing job you have been providing educating men and women using a web site. I am sure you’ve never met any of us.

  9. 2019年1月16日09:45 | #9

    A lot of thanks for your own labor on this blog. Kim really likes managing internet research and it’s easy to understand why. Many of us know all of the powerful means you convey informative tips and tricks via this website and as well as strongly encourage contribution from other individuals on that situation so our girl has always been discovering a lot of things. Have fun with the rest of the new year. You’re carrying out a first class job.

  10. 2019年1月16日18:59 | #10

    I precisely wanted to appreciate you once again. I am not sure the things that I would have made to happen without the tactics discussed by you directly on that area of interest. It actually was a fearsome concern in my position, but being able to see a new professional technique you managed that took me to weep with happiness. Now i am happy for the advice and thus hope that you comprehend what a powerful job you were providing training the rest all through your blog. I know that you haven’t encountered all of us.

  11. 2019年1月17日05:10 | #11

    I would like to show my passion for your kind-heartedness supporting men and women who really want guidance on that concern. Your very own commitment to getting the solution all through appears to be particularly useful and have constantly permitted workers like me to reach their pursuits. Your personal useful hints and tips denotes this much a person like me and still more to my colleagues. Thanks a lot; from all of us.

  12. 2019年1月17日05:45 | #12

    I wanted to send you that tiny word to thank you so much again over the remarkable tricks you have discussed on this page. It has been wonderfully open-handed with people like you to present publicly what a few individuals would have made available as an e-book in making some cash for their own end, principally considering that you could have tried it if you wanted. The suggestions as well worked to become fantastic way to recognize that most people have a similar eagerness really like my very own to know the truth more on the subject of this issue. I believe there are some more fun opportunities in the future for folks who go through your blog.

  13. 2019年1月17日13:04 | #13

    A lot of thanks for your own effort on this website. My niece loves working on investigation and it is easy to understand why. My partner and i learn all regarding the dynamic method you give both interesting and useful items through your web site and even improve contribution from other ones on the area of interest and our girl is always studying a lot of things. Take pleasure in the remaining portion of the new year. You’re carrying out a great job.

  14. 2019年1月17日23:01 | #14

    I simply wished to thank you so much again. I’m not certain the things that I might have sorted out without the entire tactics documented by you on my industry. It was the scary setting in my position, but looking at your specialized approach you managed the issue forced me to cry with fulfillment. Now i am happier for your information and believe you are aware of a great job you’re carrying out educating the rest using a site. Probably you’ve never got to know all of us.

  15. 2019年1月18日07:03 | #15

    I’m also commenting to make you understand what a perfect experience my cousin’s girl undergone reading your blog. She came to understand a wide variety of things, including how it is like to possess a great coaching mindset to get other individuals really easily comprehend several problematic issues. You truly did more than readers’ desires. I appreciate you for rendering these necessary, safe, revealing as well as fun thoughts on this topic to Tanya.

  16. 2019年1月18日14:25 | #16

    I actually wanted to make a small message so as to thank you for these amazing items you are posting at this site. My considerable internet research has at the end been honored with sensible points to write about with my friends. I ‘d say that we website visitors actually are quite endowed to live in a fantastic network with so many lovely professionals with beneficial hints. I feel extremely privileged to have seen the web site and look forward to tons of more fabulous times reading here. Thanks a lot once again for all the details.

  17. 2019年1月18日23:58 | #17

    I truly wanted to develop a quick message so as to say thanks to you for these wonderful tips and tricks you are writing on this site. My long internet search has at the end been recognized with pleasant points to go over with my relatives. I would assume that many of us readers are undoubtedly lucky to be in a decent community with many special people with very beneficial solutions. I feel very blessed to have seen the webpage and look forward to really more exciting minutes reading here. Thanks a lot again for all the details.

  18. 2019年1月19日07:09 | #18

    Thanks a lot for giving everyone an exceptionally memorable opportunity to read articles and blog posts from this web site. It is usually very nice plus packed with a great time for me personally and my office colleagues to visit your web site at the very least 3 times in a week to learn the newest guidance you have got. Not to mention, I’m usually amazed with your extraordinary tactics served by you. Some 3 ideas in this posting are clearly the most beneficial we have had.

  19. 2019年1月19日14:50 | #19

    I’m commenting to make you understand of the amazing encounter my wife’s daughter experienced checking your site. She noticed some details, most notably what it’s like to possess an incredible helping character to make a number of people completely completely grasp chosen impossible topics. You actually did more than people’s desires. Thanks for delivering the essential, dependable, informative and even cool guidance on your topic to Lizeth.

  20. 2019年1月20日01:03 | #20

    Thanks so much for giving everyone an exceptionally marvellous chance to read articles and blog posts from this website. It really is very beneficial and also packed with fun for me personally and my office colleagues to search the blog nearly thrice in one week to read the fresh stuff you will have. And lastly, I’m so usually amazed with the striking information you give. Some two points in this post are basically the finest I’ve had.

  21. 2019年1月20日01:04 | #21

    I must show appreciation to you for bailing me out of this type of condition. After researching through the the web and seeing ideas which are not productive, I figured my life was gone. Being alive without the answers to the difficulties you’ve resolved all through your main guide is a crucial case, as well as those which could have in a negative way damaged my career if I had not encountered your website. Your own skills and kindness in handling every part was useful. I’m not sure what I would have done if I hadn’t come across such a stuff like this. I can also now look ahead to my future. Thanks a lot so much for this skilled and effective help. I will not think twice to endorse your blog to anyone who should get tips on this issue.

  22. 2019年1月20日07:53 | #22

    I precisely wanted to thank you very much once again. I’m not certain what I would’ve worked on without the type of methods documented by you over this subject matter. It had become a troublesome case in my circumstances, but encountering your professional approach you managed that made me to weep for joy. I will be thankful for your support and even hope you really know what a powerful job you were getting into educating men and women all through your web blog. I know that you’ve never met all of us.

  23. 2019年1月20日16:34 | #23

    Thank you so much for giving everyone an extremely brilliant chance to read articles and blog posts from this blog. It can be very brilliant and as well , jam-packed with fun for me personally and my office mates to search your website minimum three times every week to read through the latest things you have got. Of course, I’m so certainly motivated with all the terrific guidelines you serve. Some two areas in this posting are undoubtedly the most efficient we have all had.

  24. 2019年1月21日02:08 | #24

    I simply desired to say thanks once more. I do not know what I would’ve followed without these recommendations contributed by you about such a concern. Certainly was an absolute depressing problem in my opinion, nevertheless observing a new specialised technique you managed that forced me to weep for joy. I will be happy for this work and then trust you really know what a powerful job you’re accomplishing instructing the mediocre ones using your site. More than likely you haven’t got to know any of us.

  25. 2019年1月21日08:39 | #25

    Thanks so much for giving everyone such a nice chance to discover important secrets from this web site. It’s usually so amazing and full of amusement for me and my office fellow workers to search the blog no less than thrice every week to learn the latest things you have. Not to mention, I am just always fascinated with the attractive hints you give. Certain two ideas on this page are ultimately the most beneficial we have all ever had.

  26. 2019年1月21日15:55 | #26

    I want to convey my love for your generosity giving support to those people who really need help with this theme. Your personal dedication to passing the solution all-around became certainly insightful and has in every case empowered guys and women like me to realize their pursuits. Your amazing helpful guide implies much a person like me and far more to my office colleagues. With thanks; from each one of us.

  27. 2019年1月22日18:30 | #27

    I and also my friends have already been checking the good tricks on your web blog then unexpectedly got a horrible suspicion I had not thanked the web site owner for those tips. All of the women appeared to be for that reason very interested to read them and have without a doubt been making the most of these things. Many thanks for indeed being so kind and also for finding varieties of impressive useful guides most people are really desperate to be aware of. Our honest regret for not saying thanks to you sooner.

  1. 本文目前尚无任何 trackbacks 和 pingbacks.

无觅相关文章插件,快速提升流量