Crawler changelog

Changelog History

• v1.1.1 Changes

  May 15, 2020

  • ⚡️ [Improved] Updated floki and other dependencies

• v1.1.0 Changes

  February 25, 2019

  • [Added] :modifier option
  • [Added] :encode_uri option
  • 🛠 [Improved] Varies small fixes and improvements

• v1.0.0 Changes

  August 30, 2017

  • [Added] Pause / resume / stop Crawler
  • 🛠 [Improved] Varies small fixes and improvements

• v0.4.0 Changes

  August 28, 2017

  • [Added] :scraper option to allow scraping content
  • 🛠 [Improved] Varies small fixes and improvements

• v0.3.1 Changes

  August 28, 2017

  • [Improved] Crawler.Store.DB now stores the opts meta data
  • 📚 [Improved] Code documentation
  • 🛠 [Improved] Varies small fixes and improvements

• v0.3.0 Changes

  August 26, 2017

  • [Added] :retrier option to allow custom fetch retrying logic
  • [Added] :url_filter option to allow custom url filtering logic
  • 📜 [Improved] Parser is now more stable and skips unparsable files
  • 🛠 [Improved] Varies small fixes and improvements

• v0.2.0 Changes

  August 21, 2017

  • 👷 [Added] :workers option
  • [Added] :interval option
  • ⏱ [Added] :timeout option
  • [Added] :user_agent option
  • [Added] :save_to option
  • 🍱 [Added] :assets option
  • 📜 [Added] :parser option to allow custom parsing logic
  • [Improved] Renamed :max_levels to :max_depths
  • 🛠 [Improved] Varies small fixes and improvements

• v0.1.0 Changes

  July 29, 2017

  • [Added] A semi-functioning prototype
  • [Added] Finished the very basic crawling function
  • [Added] :max_levels option