ImdbLib 0.0.2

dotnet add package ImdbLib --version 0.0.2
NuGet\Install-Package ImdbLib -Version 0.0.2
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="ImdbLib" Version="0.0.2" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add ImdbLib --version 0.0.2
#r "nuget: ImdbLib, 0.0.2"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install ImdbLib as a Cake Addin
#addin nuget:?package=ImdbLib&version=0.0.2

// Install ImdbLib as a Cake Tool
#tool nuget:?package=ImdbLib&version=0.0.2

Intro

Library and LINQPad scripts to gather, store and update data about movies in IMDB

 

Usage

using var imdb = new ImdbScraper(opt =>
{
  // Setup options
  opt.DataFolder = @"C:\IMDBData";

  // Set this to true for debugging
  // this will make the scraper work off a much smaller dataset
  opt.DbgUseSmallDatasets = true;
});
await imdb.Init();
imdb.Start();
imdb.Stop();

Run scrape.linq in LINQPad to run the scraping

 

Data flow and options

Init()

Step 1: Get IMDB dataset

datasets\title.basics.tsv This dataset is provided by IMDB and represents basic info about all the titles. If it's not present or if our copy is older than opt.DatasetRefreshPeriod (default 60 days) we download it.

Step 2: Extract dev dataset

datasets-dev\title.basics.tsv If you set opt.DbgUseSmallDatasets, it will extract a much smaller dataset (1000 titles) to use in the rest of the data flow.

Step 3: Compile titles

datasets\titles.json (or datasets-dev\titles.json if opt.DbgUseSmallDatasets is set)

  • Applies a filter specified in opt.TitleFilter to restrict the movies we should consider
  • Compiles the cleaned up results in this file

This file is regenerated if either:

  • it doesn't exist
  • the IMDB dataset was refreshed
  • the user sets opt.RefreshTitleFilter

The default filter is:

opt.TitleFilter = e =>
  !e.IsAdult &&
  e.Type == "movie" &&
  e.StartYear is >= 1970;

Note: the filter needs to discard the titles with a null StartYear for the rest of the data flow to work correctly.

These titles represent the list of all the movies we want to scrape

 

Start()

Now we have the list of titles we're interested in, we will scrape the info for them. Start() kicks off the process and is controlled by these options:

  • opt.ScrapeBatchSize the number of movies to scrape before saving (default 64)
  • opt.ScrapeParallelism determines how many movies we scrape in parallel (default 4)

There is a file to keep track of the state of each movie: scraping\title-states.json

And the movies are grouped by year in files named like this: scraping\1988.json

Note: if using opt.DbgUseSmallDatasets the files will be in the scraping-dev\ folder.

If we detect that IMDB rate limits us, we will pause for 10min.

 

LINQPad scripts

scrape.linq

Run the scraper

scrape-dbg.linq

Used to debug / tweak the scraping. It has an exact copy of the HtmlScraper class. The different functions are well documented in the script.

finder.linq

Used to query the films and nicely display the results.

Use this to find good movies!

Product Compatible and additional computed target framework versions.
.NET net7.0 is compatible.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
0.0.2 194 11/15/2022