通过模拟人类访问网页的浏览行为,提取定向爬行子页面集限定爬虫的爬行方向;引入页面继承关系,并通过爬行条目的属性继承实现跨页面复合对象的数据关联关系;设计实现了支持深度定向采集的通用爬行流程。面向天涯热帖的舆情采集实验结果表明:该方法可以在整体处理流程不变的前提下,实现复杂对象的数据采集,并具有较高的采集效率。 By simulating the browsing behavior of the human visited webpage, the crawling direction of the limited crawler is extracted from the crawling sub-page set. The page inheritance relationship is introduced, and the data association of the cross-page composite object is inherited through the properties of the crawling item. Universal crawling process. The result of public opinion collecting experiment for the End of the World hot posts shows that this method can realize the data acquisition of complex objects and has high collection efficiency under the premise of the same processing flow.