scrape alternatives and similar packages
Based on the "HTTP" category.
Alternatively, view scrape alternatives based on common mentions on social networks and blogs.
-
mint
Functional HTTP client for Elixir with support for HTTP/1 and HTTP/2 🌱 -
PlugAttack
A plug building toolkit for blocking and throttling abusive requests -
spell
Spell is a Web Application Messaging Protocol (WAMP) client implementation in Elixir. WAMP is an open standard WebSocket subprotocol that provides two application messaging patterns in one unified protocol: Remote Procedure Calls + Publish & Subscribe: http://wamp.ws/ -
web_socket
An exploration into a stand-alone library for Plug applications to easily adopt WebSockets. -
http_proxy
http proxy with Elixir. wait request with multi port and forward to each URIs -
explode
An easy utility for responding with standard HTTP/JSON error payloads in Plug- and Phoenix-based applications -
Mechanize
Build web scrapers and automate interaction with websites in Elixir with ease! -
ivar
Ivar is an adapter based HTTP client that provides the ability to build composable HTTP requests. -
fuzzyurl
An Elixir library for non-strict parsing, manipulation, and wildcard matching of URLs. -
SpiderMan
SpiderMan,a base-on Broadway fast high-level web crawling & scraping framework for Elixir. -
http_digex
HTTP Digest Auth Library to create auth header to be used with HTTP Digest Authentication -
Ralitobu.Plug
Elixir Plug for Ralitobu, the Rate Limiter with Token Bucket algorithm
Clean code begins in your IDE with SonarLint
Do you think we are missing an alternative of scrape or a related project?
Popular Comparisons
README
Scrape
Structured Data extraction from common web resources, using information-retrieval techniques. See the docs
Installation
The package can be installed by adding scrape
to your list of dependencies in mix.exs
:
def deps do
[
{:scrape, "~> 3.0.0"}
]
end
Known Issues
- This package uses an outdated version of
httpoison
because ofkeepcosmos/readability
. You can override this in your app withoverride: true
and everything should work. - The current version 3.X is a complete rewrite from scratch, so some new issues might occur and the API has changed. Please provide some URL to a HTML/Feed document when submitting issues, so I can look into it for bugfixing.
Usage
Scrape.domain!(url)
-> get structured data of a domain-type url (like https://bbc.com)Scrape.feed!(url)
-> get structured data of a RSS/Atom feedScrape.article!(url)
-> get structured data of an article-type url
License
LGPLv3. You can use this package any way you want (including commercially), but I want bugfixes and improvements to flow back into this package for everyone's benefit.
*Note that all licence references and agreements mentioned in the scrape README section above
are relevant to that project's source code only.