Earl.Crawler
0.0.0-alpha.0.111
dotnet add package Earl.Crawler --version 0.0.0-alpha.0.111
NuGet\Install-Package Earl.Crawler -Version 0.0.0-alpha.0.111
<PackageReference Include="Earl.Crawler" Version="0.0.0-alpha.0.111" />
paket add Earl.Crawler --version 0.0.0-alpha.0.111
#r "nuget: Earl.Crawler, 0.0.0-alpha.0.111"
// Install Earl.Crawler as a Cake Addin #addin nuget:?package=Earl.Crawler&version=0.0.0-alpha.0.111&prerelease // Install Earl.Crawler as a Cake Tool #tool nuget:?package=Earl.Crawler&version=0.0.0-alpha.0.111&prerelease
<h1 align="center">earl</h1>
<div align="center">
Looking for URLs in your area.
</div>
Earl is a suite of APIs for developing url crawlers & web scrapers driven by a middleware pattern similar to, and strongly influenced by, ASP.NET Core.
Basic Usage
var services = new ServiceCollection()
.AddEarlCrawler()
.AddEarlJsonPersistence()
.BuildServiceProvider();
var crawler = services.GetService<IEarlCrawler>();
var options = CrawlerOptionsBuilder.CreateDefault()
.BatchSize( 50 )
.MaxRequestCount( 500 )
.On<CrawlUrlResultEvent>(
( CrawlUrlResultEvent e, CancellationToken cancellation ) =>
{
Console.WriteLine( $"Crawled {e.Result.Url}" );
return default;
}
)
.Timeout( TimeSpan.FromMinutes( 30 ) )
.Use(
( CrawlUrlContext context, CrawlUrlDelegate next ) =>
{
Console.WriteLine( $"Executing delegate middleware while crawling {context.Url}" );
return next( context );
}
)
.PersistTo( persist => persist.ToJson( json => json.Destination(...) ) )
.Build();
await crawler.CrawlAsync( new Uri(...), options );
Documentation
Documentation can be find within the READMEs of the sub-directories representing the conceptual components of Earl:
All public APIs should contain thorough XML (triple slash) comments.
Something missing, still have questions? Please open an Issue or submit a PR!
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
-
net6.0
- Earl.Crawler.Abstractions (>= 0.0.0-alpha.0.111)
- Earl.Crawler.Events (>= 0.0.0-alpha.0.111)
- Earl.Crawler.Middleware (>= 0.0.0-alpha.0.111)
- Earl.Crawler.Middleware.Html (>= 0.0.0-alpha.0.111)
- Earl.Crawler.Middleware.Http (>= 0.0.0-alpha.0.111)
- Earl.Crawler.Middleware.UrlScraping (>= 0.0.0-alpha.0.111)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.