Tutorial StormCrawler 1.5 + Elasticsearch 5.3

What people said

Recommends Julien Nioche · 2019-11 · Stack Overflow · on “how to store content and page description to ElasticSearch”

This IndexerBolt does not index the documents to Elasticsearch, it is used for debugging and sends the content to the console. The one you want [LINK] is in the ES module. The part of the schema you copied deals with the status of the URLs, not their content. BTW you definitely don't want to index the content field as keywords . You'll need to change the mapping for the content index, add store true to the content field [LINK] , see also ES documentation [LINK] . The video tutorial [this video] for SC 1.5 is a bit dated but should help you understand the basics.

Recommends Daniel Agudelo Cruz · 2019-12 · Stack Overflow · on “What values/fields can I get from stormcrawler?”

I am using Stormcrawler 1.15, ElasticSearch 7.5, and followed this tutorial to get SC up and running: [this video] In my crawler-conf.yaml I have: # indexer.md.filter: "someKey=aValue" indexer.url.fieldname: "url" indexer.text.fieldname: "content" indexer.canonical.name: "canonical" indexer.md.mapping: - parse.title=title - parse.keywords=keywords - parse.description=description - domain=domain This suggests that I already have description and keywords, but as of now, I get in the documents of the 'content' index the fields: url, content, domain, title. For instance: { "_index": "content", "_type": "_doc", "_id": "c31104689b7ab4e8152f70b755049e7060da3457f06d64993667df28f7be6811", "_score": 1, "_s

Cites Julien Nioche · 2017-08 · Stack Overflow · on “Tuning Storm-Crawler to fully use available resources”

You could use Kibana or Grafana to visualise the metrics generated by StormCrawler, see tutorial [this video] . This would give you some insights into the performance. Also, the Storm UI will tell you of bottlenecks at the component level. You could use more than 2 shards for the status index and have the corresponding number of spout instances. This would increase the parallelism. Do you follow outlinks for the webpages or does the size of your index remain constant? 50M urls is not very much so I don't expect that ES would be super busy. Have you tried using the AggregationSpout instead? The CollapsingSpout is pretty new + it is probably better to use it with a bucket size of 1 as I think it em

Cites Jonas Pohlmann · 2018-04 · Stack Overflow · on “Stormcrawler not fetching/indexing pages for elasticsearch”

I am using the Stormcrawler with the Elasticsearch example [this video] and no pages are shown with the FETCHED status in Kibana while crawling the webpage [LINK] Still on the console the webpages appear to be fetched and parsed 48239 [Thread-26-fetcher-executor[3 3]] INFO c.d.s.b.FetcherBolt - [Fetcher #3] Threads : 0 queues : 1 in_queues : 1 48341 [FetcherThread #7] INFO c.d.s.b.FetcherBolt - [Fetcher #3] Fetched [LINK] with status 200 in msec 86 48346 [Thread-46-parse-executor[5 5]] INFO c.d.s.b.JSoupParserBolt - Parsing : starting [LINK] 48362 [Thread-46-parse-executor[5 5]] INFO c.d.s.b.JSoupParserBolt - Parsed [LINK] in 13 msec Also it seems that the index of Elasticsearch gets some items e

Cites Juan Diego Chaves Sanguino · 2019-12 · Stack Overflow · on “Add field to crawled content with StormCrawler (and Elasticsearch)”

I have followed the following tutorial for crawling content with stormcrawler and then store it in elasticsearch: [this video] . However, I would like to add to every document the date it was crawled. Can anyone tell me how this can be done? In general, how can I change the fields of the crawled content? Thanks in advance