Scraper for serienstream.to (s.to) + series lister
- Migrated sto scraper to Playwright Firefox (puppeteer 24 is Chromium-only and DDOS-Guard blocks Chromium) - Handle tab-scoped auth by reusing the authenticated page - Add npm run list:series to summarize scraped data (series + watched eps)
This commit is contained in:
commit
75221c9747
14 changed files with 3311 additions and 0 deletions
7
.env.example
Normal file
7
.env.example
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
# Login credentials for s.to and aniworld.to
|
||||||
|
PAGE_USERNAME=your_email@example.com
|
||||||
|
PAGE_PASSWORD=your_password
|
||||||
|
|
||||||
|
# Browser settings
|
||||||
|
HEADLESS=true
|
||||||
|
DONT_LOAD_STYLES=true
|
||||||
19
.gitignore
vendored
Normal file
19
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,19 @@
|
||||||
|
# Environment variables
|
||||||
|
.env
|
||||||
|
|
||||||
|
# Dependencies
|
||||||
|
node_modules/
|
||||||
|
|
||||||
|
# Data files
|
||||||
|
data/*.json
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.idea/
|
||||||
|
.vscode/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
.ai/
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
21
LICENSE.md
Normal file
21
LICENSE.md
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2023 Kamikaze
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
219
README.md
Normal file
219
README.md
Normal file
|
|
@ -0,0 +1,219 @@
|
||||||
|
# STO Scraper v2
|
||||||
|
|
||||||
|
A modernized web scraper for **s.to** and **aniworld.to** that extracts your viewing history with a beautiful standalone viewer.
|
||||||
|
|
||||||
|
## ✨ What's New in v2
|
||||||
|
|
||||||
|
- **🔄 Split Scrapers**: Separate scrapers for s.to and aniworld.to (different layouts)
|
||||||
|
- **📱 Standalone Viewer**: Self-contained HTML file with Vue 3 + Vuetify 3
|
||||||
|
- **🌙 Dark Mode**: Beautiful dark theme by default (toggleable)
|
||||||
|
- **📊 Better Data**: Metadata included (scrape date, site, stats)
|
||||||
|
- **🎯 TypeScript**: Fully typed codebase with proper structure
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
### 1. Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone the repository
|
||||||
|
git clone https://github.com/Kamiikaze/sto-scraper.git
|
||||||
|
cd sto-scraper
|
||||||
|
|
||||||
|
# Install dependencies
|
||||||
|
npm install
|
||||||
|
|
||||||
|
# Set up your credentials
|
||||||
|
cp .env.example .env
|
||||||
|
# Edit .env with your credentials
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Configure Environment
|
||||||
|
|
||||||
|
Edit `.env` file:
|
||||||
|
|
||||||
|
```env
|
||||||
|
PAGE_USERNAME=your_email@example.com
|
||||||
|
PAGE_PASSWORD=your_password
|
||||||
|
|
||||||
|
HEADLESS=true # Set to false to see browser during scraping
|
||||||
|
DONT_LOAD_STYLES=true # Speeds up scraping
|
||||||
|
DO_SCREENSHOTS=false # Set to true to save debug screenshots
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Scrape Your History
|
||||||
|
|
||||||
|
### Note: If you have different credentials for s.to and aniworld.to, you need to change the credentials in `.env` before running the scraper. (Will be fixed soon)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Scrape s.to (all pages)
|
||||||
|
npm run scrape:sto
|
||||||
|
|
||||||
|
# Scrape aniworld.to (all pages)
|
||||||
|
npm run scrape:aniworld
|
||||||
|
|
||||||
|
# Scrape first 5 pages only (for testing)
|
||||||
|
npm run scrape:sto:limit
|
||||||
|
npm run scrape:aniworld:limit
|
||||||
|
```
|
||||||
|
Or use the interactive `scrape.bat`
|
||||||
|
|
||||||
|
Results are saved to `data/` as JSON files.
|
||||||
|
|
||||||
|
### 4. View Your History
|
||||||
|
|
||||||
|
Simply **run `npm run webviewer` and open http://localhost:8080 in your browser**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📁 Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
sto-scraper-v2/
|
||||||
|
├── src/
|
||||||
|
│ ├── types.ts # Shared TypeScript types
|
||||||
|
│ └── scrapers/
|
||||||
|
│ ├── scraper-utils.ts # Shared scraping utilities
|
||||||
|
│ ├── scraper-sto.ts # s.to scraper
|
||||||
|
│ └── scraper-aniworld.ts # aniworld.to scraper
|
||||||
|
├── data/ # Viewer + Scraped JSON files (gitignored)
|
||||||
|
│ └── index.html # Standalone viewer (Vue 3 + Vuetify 3)
|
||||||
|
├── package.json
|
||||||
|
├── tsconfig.json
|
||||||
|
├── .env.example
|
||||||
|
└── README.md
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🛠️ How It Works
|
||||||
|
|
||||||
|
### Scrapers
|
||||||
|
|
||||||
|
Each site has its own scraper class:
|
||||||
|
|
||||||
|
- **`scraper-sto.ts`**
|
||||||
|
- **`scraper-aniworld.ts`**
|
||||||
|
|
||||||
|
Both scrapers:
|
||||||
|
1. Log in with your credentials
|
||||||
|
2. Navigate through all history pages
|
||||||
|
3. Extract series, seasons, episodes, and watch times
|
||||||
|
4. Save structured JSON to `data/`
|
||||||
|
|
||||||
|
### Data Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"totalMovies": 42,
|
||||||
|
"totalEpisodesCount": 4986,
|
||||||
|
"site": "aniworld.to",
|
||||||
|
"scrapedAt": "2026-04-21T10:30:00.000Z",
|
||||||
|
"movieTitles": {
|
||||||
|
"Attack on Titan": {
|
||||||
|
"seasonCount": 4,
|
||||||
|
"totalEpisodesCount": 87,
|
||||||
|
"seasons": {
|
||||||
|
"1": {
|
||||||
|
"episodeCount": 25,
|
||||||
|
"episodes": {
|
||||||
|
"1": {
|
||||||
|
"title": "To You, in 2000 Years",
|
||||||
|
"seenAt": "21.04.2026 15:30:00 Uhr vor 2 Tagen"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Viewer
|
||||||
|
|
||||||
|
The `index.html` is a **completely standalone file**:
|
||||||
|
|
||||||
|
- Uses CDN-hosted Vue 3 + Vuetify 3
|
||||||
|
- Reads the `*_latest.json` files
|
||||||
|
- Dark mode by default
|
||||||
|
- Fully responsive design
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎨 Viewer Features
|
||||||
|
|
||||||
|
### Series Display
|
||||||
|
- Episode tables with:
|
||||||
|
- Episode number badges
|
||||||
|
- Episode titles
|
||||||
|
- Watch timestamps
|
||||||
|
- Stats chips showing:
|
||||||
|
- Total series count
|
||||||
|
- Total episodes count
|
||||||
|
- Site name
|
||||||
|
- Scrape date
|
||||||
|
|
||||||
|
### Theme
|
||||||
|
- Dark mode by default (easy on the eyes)
|
||||||
|
- Toggle to light mode via sun/moon icon
|
||||||
|
- Consistent Material Design 3 styling
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Advanced Usage
|
||||||
|
|
||||||
|
### Custom Scrape Limits
|
||||||
|
|
||||||
|
Run scrapers with custom page limits:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Scrape first 10 pages of s.to
|
||||||
|
npx ts-node src/scrapers/scraper-sto.ts 10
|
||||||
|
|
||||||
|
# Scrape first 20 pages of aniworld.to
|
||||||
|
npx ts-node src/scrapers/scraper-aniworld.ts 20
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🐛 Troubleshooting
|
||||||
|
|
||||||
|
### "Login failed"
|
||||||
|
- Check your credentials in `.env`
|
||||||
|
- Try with `HEADLESS=false` to see what's happening
|
||||||
|
- Site might have captcha - solve it manually with headless=false
|
||||||
|
|
||||||
|
### "No data in viewer"
|
||||||
|
- Make sure JSON files are in `data/` folder
|
||||||
|
- Check browser console for errors
|
||||||
|
|
||||||
|
### "Module not found"
|
||||||
|
- Run `npm install` to install dependencies
|
||||||
|
- Make sure you're in the project directory
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📝 License
|
||||||
|
|
||||||
|
MIT License - See [LICENSE](LICENSE) for details.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🙏 Credits
|
||||||
|
|
||||||
|
- Original project by **Kamiikaze**
|
||||||
|
- Modernized with Vue 3, Vuetify 3, and TypeScript
|
||||||
|
- Built with ❤️ for tracking your anime/series history
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔮 Future Ideas
|
||||||
|
|
||||||
|
- [ ] Merge multiple scrapes into unified view
|
||||||
|
- [ ] Export to CSV/Excel
|
||||||
|
- [ ] Support exports to other platforms (e.g. MyAnimeList, IMDB)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Enjoy your modernized viewing history tracker! 🎉**
|
||||||
424
data/index.html
Normal file
424
data/index.html
Normal file
|
|
@ -0,0 +1,424 @@
|
||||||
|
<!doctype html>
|
||||||
|
<html lang="de">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
|
<title>Watch History</title>
|
||||||
|
|
||||||
|
<!-- Vuetify 3 CSS -->
|
||||||
|
<link
|
||||||
|
href="https://cdn.jsdelivr.net/npm/vuetify@4.0.5/dist/vuetify.min.css"
|
||||||
|
rel="stylesheet"
|
||||||
|
/>
|
||||||
|
<!-- Material Design Icons -->
|
||||||
|
<link
|
||||||
|
href="https://cdn.jsdelivr.net/npm/@mdi/font@7.4.47/css/materialdesignicons.min.css"
|
||||||
|
rel="stylesheet"
|
||||||
|
/>
|
||||||
|
|
||||||
|
<style>
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
font-family: "Roboto", sans-serif;
|
||||||
|
}
|
||||||
|
|
||||||
|
.search-bar-sticky {
|
||||||
|
position: sticky;
|
||||||
|
top: 48px;
|
||||||
|
z-index: 100;
|
||||||
|
/*noinspection CssUnresolvedCustomProperty*/
|
||||||
|
background: rgb(var(--v-theme-surface));
|
||||||
|
border-bottom: 1px solid
|
||||||
|
rgba(var(--v-border-color), var(--v-border-opacity));
|
||||||
|
}
|
||||||
|
|
||||||
|
.episode-list .v-row:nth-child(even) {
|
||||||
|
background-color: rgba(255, 255, 255, 0.025);
|
||||||
|
}
|
||||||
|
.episode-list .v-row:nth-child(odd) {
|
||||||
|
background-color: rgba(255, 255, 255, 0.01);
|
||||||
|
}
|
||||||
|
|
||||||
|
.episode-list .v-row {
|
||||||
|
transition: background-color 0.15s;
|
||||||
|
}
|
||||||
|
|
||||||
|
.episode-list .v-row:hover {
|
||||||
|
background-color: rgba(255, 255, 255, 0.05);
|
||||||
|
}
|
||||||
|
|
||||||
|
.stats-chip {
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
.no-data-container {
|
||||||
|
min-height: 60vh;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Panel title: truncate long titles, keep chips visible */
|
||||||
|
.panel-title-inner {
|
||||||
|
min-width: 0;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 4px;
|
||||||
|
width: 100%;
|
||||||
|
}
|
||||||
|
|
||||||
|
.panel-title-text {
|
||||||
|
min-width: 0;
|
||||||
|
flex: 1 1 200px;
|
||||||
|
overflow: hidden;
|
||||||
|
text-wrap-style: balance;
|
||||||
|
line-height: 1.4;
|
||||||
|
}
|
||||||
|
|
||||||
|
.panel-title-chips {
|
||||||
|
flex-shrink: 0;
|
||||||
|
display: flex;
|
||||||
|
gap: 8px;
|
||||||
|
align-items: center;
|
||||||
|
/* keep chips away from expansion arrow */
|
||||||
|
margin-left: 8px;
|
||||||
|
margin-right: 8px;
|
||||||
|
}
|
||||||
|
@media (max-width: 600px) {
|
||||||
|
.panel-title-chips {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
|
||||||
|
<body>
|
||||||
|
<div id="app">
|
||||||
|
<v-app>
|
||||||
|
<v-app-bar color="primary" density="compact">
|
||||||
|
<template v-slot:prepend>
|
||||||
|
<v-icon>mdi-movie-open</v-icon>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<v-app-bar-title>
|
||||||
|
Watch History
|
||||||
|
<small>
|
||||||
|
|
|
||||||
|
<a
|
||||||
|
class="text-black"
|
||||||
|
href="https://github.com/Kamiikaze/sto-scraper"
|
||||||
|
>Kamiikaze/sto-scraper</a
|
||||||
|
></small
|
||||||
|
>
|
||||||
|
</v-app-bar-title>
|
||||||
|
|
||||||
|
<template v-slot:append>
|
||||||
|
<v-btn icon @click="toggleTheme">
|
||||||
|
<v-icon
|
||||||
|
>{{ isDark ? 'mdi-white-balance-sunny' : 'mdi-weather-night'
|
||||||
|
}}</v-icon
|
||||||
|
>
|
||||||
|
</v-btn>
|
||||||
|
</template>
|
||||||
|
</v-app-bar>
|
||||||
|
|
||||||
|
<v-main>
|
||||||
|
<!-- Sticky Bar -->
|
||||||
|
<div class="search-bar-sticky">
|
||||||
|
<v-container>
|
||||||
|
<v-row align="center" no-gutters class="gap-2 py-1">
|
||||||
|
<!-- List Type Toggle -->
|
||||||
|
<v-col cols="auto" class="d-flex align-center mr-2">
|
||||||
|
<v-btn-toggle
|
||||||
|
v-model="listType"
|
||||||
|
color="primary"
|
||||||
|
density="comfortable"
|
||||||
|
mandatory
|
||||||
|
border
|
||||||
|
>
|
||||||
|
<v-btn value="serie">Serien</v-btn>
|
||||||
|
<v-btn value="anime">Animes</v-btn>
|
||||||
|
</v-btn-toggle>
|
||||||
|
</v-col>
|
||||||
|
|
||||||
|
<!-- Search Field -->
|
||||||
|
<v-col class="d-flex align-center">
|
||||||
|
<v-text-field
|
||||||
|
v-model="searchQuery"
|
||||||
|
prepend-inner-icon="mdi-magnify"
|
||||||
|
placeholder="Titel suchen…"
|
||||||
|
density="compact"
|
||||||
|
variant="outlined"
|
||||||
|
hide-details
|
||||||
|
clearable
|
||||||
|
class="w-100"
|
||||||
|
></v-text-field>
|
||||||
|
</v-col>
|
||||||
|
</v-row>
|
||||||
|
|
||||||
|
<v-row
|
||||||
|
class="d-flex justify-center align-center mt-3 ga-2"
|
||||||
|
v-if="filteredSeries.length > 0"
|
||||||
|
>
|
||||||
|
<!-- Stats Chips (hidden below lg) -->
|
||||||
|
<v-col cols="auto">
|
||||||
|
<v-chip class="stats-chip" color="secondary" variant="flat">
|
||||||
|
<v-icon start>mdi-web</v-icon>
|
||||||
|
{{ currentData.site || 'Unknown' }}
|
||||||
|
</v-chip>
|
||||||
|
</v-col>
|
||||||
|
<v-col cols="auto">
|
||||||
|
<v-chip class="stats-chip" color="primary" variant="flat">
|
||||||
|
<v-icon start>mdi-television-classic</v-icon>
|
||||||
|
{{ currentData.totalMovies }} Shows
|
||||||
|
</v-chip>
|
||||||
|
</v-col>
|
||||||
|
<v-col cols="auto">
|
||||||
|
<v-chip class="stats-chip" color="warning" variant="flat">
|
||||||
|
<v-icon start>mdi-television-classic</v-icon>
|
||||||
|
{{ currentData.totalEpisodesCount }} Episoden
|
||||||
|
</v-chip>
|
||||||
|
</v-col>
|
||||||
|
<v-col cols="auto">
|
||||||
|
<v-chip
|
||||||
|
class="stats-chip"
|
||||||
|
color="info"
|
||||||
|
variant="flat"
|
||||||
|
v-if="currentData.scrapedAt"
|
||||||
|
>
|
||||||
|
<v-icon start>mdi-clock</v-icon>
|
||||||
|
{{ formatDate(currentData.scrapedAt) }}
|
||||||
|
</v-chip>
|
||||||
|
</v-col>
|
||||||
|
</v-row>
|
||||||
|
</v-container>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Content Area -->
|
||||||
|
<v-container class="py-6">
|
||||||
|
<!-- No data state -->
|
||||||
|
<div
|
||||||
|
v-if="!loading && filteredSeries.length === 0"
|
||||||
|
class="no-data-container"
|
||||||
|
>
|
||||||
|
<v-card class="text-center pa-8" elevation="2" max-width="500">
|
||||||
|
<v-icon size="80" color="primary" class="mb-4"
|
||||||
|
>mdi-alert-circle</v-icon
|
||||||
|
>
|
||||||
|
<h2 class="text-h5 mb-4">
|
||||||
|
{{ searchQuery ? 'Keine Treffer' : 'Keine Daten gefunden' }}
|
||||||
|
</h2>
|
||||||
|
<p class="text-body-1 mb-4" v-if="!searchQuery">
|
||||||
|
Überprüfe, ob im <code>./data</code> Ordner eine
|
||||||
|
<code>{seite}_latest.json</code> Datei vorhanden ist.
|
||||||
|
</p>
|
||||||
|
<p class="text-body-1 mb-4" v-else>
|
||||||
|
Kein Titel enthält „{{ searchQuery }}".
|
||||||
|
</p>
|
||||||
|
</v-card>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Series List -->
|
||||||
|
<v-expansion-panels v-if="!loading && filteredSeries.length > 0">
|
||||||
|
<v-expansion-panel
|
||||||
|
v-for="series in filteredSeries"
|
||||||
|
:key="series.title"
|
||||||
|
elevation="3"
|
||||||
|
>
|
||||||
|
<v-expansion-panel-title>
|
||||||
|
<div class="panel-title-inner">
|
||||||
|
<v-icon class="mr-1 flex-shrink-0" color="primary"
|
||||||
|
>mdi-movie-roll</v-icon
|
||||||
|
>
|
||||||
|
<span class="panel-title-text text-h6"
|
||||||
|
>{{ series.title }}</span
|
||||||
|
>
|
||||||
|
<div class="panel-title-chips">
|
||||||
|
<v-chip size="small" color="primary" variant="outlined">
|
||||||
|
{{ series.seasonCount }} Staffel{{ series.seasonCount
|
||||||
|
!== 1 ? 'n' : '' }}
|
||||||
|
</v-chip>
|
||||||
|
<v-chip size="small" color="secondary" variant="outlined">
|
||||||
|
{{ series.totalEpisodesCount }} Episode{{
|
||||||
|
series.totalEpisodesCount !== 1 ? 'n' : '' }}
|
||||||
|
</v-chip>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</v-expansion-panel-title>
|
||||||
|
|
||||||
|
<v-expansion-panel-text>
|
||||||
|
<v-divider class="mb-4"></v-divider>
|
||||||
|
<div
|
||||||
|
class="episode-list mb-6"
|
||||||
|
v-for="season in series.seasons"
|
||||||
|
:key="season.num"
|
||||||
|
>
|
||||||
|
<h2 class="mb-2">
|
||||||
|
{{ season.num === 0 ? "Film / Specials" : `Staffel
|
||||||
|
${season.num}` }}
|
||||||
|
</h2>
|
||||||
|
<v-row
|
||||||
|
class="mt-0 py-2 px-2"
|
||||||
|
v-for="episode in season.episodes"
|
||||||
|
:key="episode.num"
|
||||||
|
>
|
||||||
|
<v-col cols="2" sm="1">{{ episode.num }}</v-col>
|
||||||
|
<v-col cols="10" sm="8">{{ episode.title }}</v-col>
|
||||||
|
<v-col cols="3" class="d-none d-sm-flex justify-end"
|
||||||
|
>{{ episode.seenAt }}</v-col
|
||||||
|
>
|
||||||
|
</v-row>
|
||||||
|
</div>
|
||||||
|
</v-expansion-panel-text>
|
||||||
|
</v-expansion-panel>
|
||||||
|
</v-expansion-panels>
|
||||||
|
|
||||||
|
<!-- Loading -->
|
||||||
|
<div v-if="loading" class="d-flex justify-center align-center">
|
||||||
|
<v-progress-circular
|
||||||
|
indeterminate
|
||||||
|
color="primary"
|
||||||
|
></v-progress-circular>
|
||||||
|
<span class="ml-4">Daten werden geladen...</span>
|
||||||
|
</div>
|
||||||
|
</v-container>
|
||||||
|
</v-main>
|
||||||
|
</v-app>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Vue 3 -->
|
||||||
|
<script src="https://cdn.jsdelivr.net/npm/vue@3.5.32/dist/vue.global.js"></script>
|
||||||
|
<!-- Vuetify 3 -->
|
||||||
|
<script src="https://cdn.jsdelivr.net/npm/vuetify@4.0.5/dist/vuetify.min.js"></script>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
const { createApp } = Vue;
|
||||||
|
const { createVuetify } = Vuetify;
|
||||||
|
|
||||||
|
const vuetify = createVuetify({
|
||||||
|
theme: {
|
||||||
|
defaultTheme: "dark",
|
||||||
|
themes: {
|
||||||
|
dark: {
|
||||||
|
colors: {
|
||||||
|
primary: "#BB86FC",
|
||||||
|
secondary: "#03DAC6",
|
||||||
|
accent: "#CF6679",
|
||||||
|
error: "#CF6679",
|
||||||
|
info: "#2196F3",
|
||||||
|
success: "#4CAF50",
|
||||||
|
warning: "#FB8C00",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
light: {
|
||||||
|
colors: {
|
||||||
|
primary: "#6200EE",
|
||||||
|
secondary: "#03DAC6",
|
||||||
|
accent: "#018786",
|
||||||
|
error: "#B00020",
|
||||||
|
info: "#2196F3",
|
||||||
|
success: "#4CAF50",
|
||||||
|
warning: "#FB8C00",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
createApp({
|
||||||
|
data() {
|
||||||
|
return {
|
||||||
|
animeList: null,
|
||||||
|
serieList: null,
|
||||||
|
processedAnimeList: [],
|
||||||
|
processedSerieList: [],
|
||||||
|
listType: "serie",
|
||||||
|
searchQuery: "",
|
||||||
|
isDark: true,
|
||||||
|
loading: true,
|
||||||
|
};
|
||||||
|
},
|
||||||
|
computed: {
|
||||||
|
currentData() {
|
||||||
|
return this.listType === "serie"
|
||||||
|
? this.serieList || {}
|
||||||
|
: this.animeList || {};
|
||||||
|
},
|
||||||
|
filteredSeries() {
|
||||||
|
const list =
|
||||||
|
this.listType === "serie"
|
||||||
|
? this.processedSerieList
|
||||||
|
: this.processedAnimeList;
|
||||||
|
if (!this.searchQuery?.trim()) return list;
|
||||||
|
const q = this.searchQuery.toLowerCase();
|
||||||
|
return list.filter((s) => s.title.toLowerCase().includes(q));
|
||||||
|
},
|
||||||
|
},
|
||||||
|
methods: {
|
||||||
|
async fetchData(url) {
|
||||||
|
try {
|
||||||
|
const response = await fetch(url);
|
||||||
|
if (!response.ok) {
|
||||||
|
throw new Error(`HTTP error! status: ${response.status}`);
|
||||||
|
}
|
||||||
|
return await response.json();
|
||||||
|
} catch (error) {
|
||||||
|
console.error("Error fetching data:", error);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
formatDate(dateString) {
|
||||||
|
return new Date(dateString).toLocaleString("de-DE", {
|
||||||
|
year: "numeric",
|
||||||
|
month: "2-digit",
|
||||||
|
day: "2-digit",
|
||||||
|
hour: "2-digit",
|
||||||
|
minute: "2-digit",
|
||||||
|
});
|
||||||
|
},
|
||||||
|
processData(raw) {
|
||||||
|
if (!raw?.movieTitles) return [];
|
||||||
|
return Object.entries(raw.movieTitles).map(([title, data]) => ({
|
||||||
|
title,
|
||||||
|
seasonCount: data.seasonCount,
|
||||||
|
totalEpisodesCount: data.totalEpisodesCount,
|
||||||
|
seasons: Object.entries(data.seasons).map(([sNum, season]) => ({
|
||||||
|
num: parseInt(sNum),
|
||||||
|
episodes: Object.entries(season.episodes).map(
|
||||||
|
([epNum, ep]) => ({
|
||||||
|
num: epNum,
|
||||||
|
title: ep.title,
|
||||||
|
seenAt: this.formatDate(ep.seenAt),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
})),
|
||||||
|
}));
|
||||||
|
},
|
||||||
|
toggleTheme() {
|
||||||
|
this.isDark = !this.isDark;
|
||||||
|
this.$vuetify.theme.global.name = this.isDark ? "dark" : "light";
|
||||||
|
},
|
||||||
|
},
|
||||||
|
async mounted() {
|
||||||
|
this.loading = true;
|
||||||
|
|
||||||
|
const [anime, serie] = await Promise.all([
|
||||||
|
this.fetchData("./aniworld.to_latest.json"),
|
||||||
|
this.fetchData("./s.to_latest.json"),
|
||||||
|
]);
|
||||||
|
|
||||||
|
this.animeList = anime;
|
||||||
|
this.serieList = serie;
|
||||||
|
this.processedAnimeList = this.processData(anime);
|
||||||
|
this.processedSerieList = this.processData(serie);
|
||||||
|
|
||||||
|
this.loading = false;
|
||||||
|
},
|
||||||
|
})
|
||||||
|
.use(vuetify)
|
||||||
|
.mount("#app");
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
1919
package-lock.json
generated
Normal file
1919
package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load diff
30
package.json
Normal file
30
package.json
Normal file
|
|
@ -0,0 +1,30 @@
|
||||||
|
{
|
||||||
|
"name": "sto-scraper",
|
||||||
|
"description": "Modernized scraper for s.to and aniworld.to with separate site support",
|
||||||
|
"version": "2.0.0",
|
||||||
|
"scripts": {
|
||||||
|
"webviewer": "http-server -p 8080 data -o",
|
||||||
|
"scrape:sto": "npx ts-node src/scrapers/scraper-sto.ts",
|
||||||
|
"scrape:aniworld": "npx ts-node src/scrapers/scraper-aniworld.ts",
|
||||||
|
"scrape:sto:limit": "npx ts-node src/scrapers/scraper-sto.ts 5",
|
||||||
|
"scrape:aniworld:limit": "npx ts-node src/scrapers/scraper-aniworld.ts 5",
|
||||||
|
"list:series": "npx ts-node src/scripts/list-series.ts"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"dotenv": "17.4.2",
|
||||||
|
"playwright": "^1.62.1",
|
||||||
|
"puppeteer": "^24.42.0",
|
||||||
|
"ts-node": "10.9.2"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/node": "25.6.0",
|
||||||
|
"http-server": "14.1.1",
|
||||||
|
"typescript": "6.0.3"
|
||||||
|
},
|
||||||
|
"repository": {
|
||||||
|
"type": "git",
|
||||||
|
"url": "https://github.com/Kamiikaze/sto-scraper.git"
|
||||||
|
},
|
||||||
|
"author": "Kamiikaze",
|
||||||
|
"license": "MIT"
|
||||||
|
}
|
||||||
12
scrape.bat
Normal file
12
scrape.bat
Normal file
|
|
@ -0,0 +1,12 @@
|
||||||
|
@echo off
|
||||||
|
set /p website=Welche Website? (aniworld oder sto):
|
||||||
|
|
||||||
|
if /i "%website%"=="aniworld" (
|
||||||
|
npm run scrape:aniworld
|
||||||
|
) else if /i "%website%"=="sto" (
|
||||||
|
npm run scrape:sto
|
||||||
|
) else (
|
||||||
|
echo Ungültige Website!
|
||||||
|
)
|
||||||
|
|
||||||
|
pause
|
||||||
180
src/scrapers/scraper-aniworld.ts
Normal file
180
src/scrapers/scraper-aniworld.ts
Normal file
|
|
@ -0,0 +1,180 @@
|
||||||
|
import { Page } from "playwright";
|
||||||
|
import { ScraperConfig, ScraperOutput, SeriesData } from "../types";
|
||||||
|
import { ScraperUtils } from "./scraper-utils";
|
||||||
|
|
||||||
|
const ANIWORLD_CONFIG: ScraperConfig = {
|
||||||
|
website: "aniworld.to",
|
||||||
|
loginUrl: "https://aniworld.to/login",
|
||||||
|
searchUrl: "https://aniworld.to/account/watched/",
|
||||||
|
selectors: {
|
||||||
|
nav: ".hosterSiteDirectNav.pagination ul",
|
||||||
|
row: "table tbody tr",
|
||||||
|
title: "td:nth-child(1) > a",
|
||||||
|
info: "td:nth-child(2) > a > strong",
|
||||||
|
time: "td:nth-child(3)",
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
export class AniworldScraper {
|
||||||
|
private utils: ScraperUtils;
|
||||||
|
private config: ScraperConfig;
|
||||||
|
private movieTitles: { [title: string]: SeriesData } = {};
|
||||||
|
private limit: number;
|
||||||
|
|
||||||
|
constructor(limit: number = 0) {
|
||||||
|
this.config = ANIWORLD_CONFIG;
|
||||||
|
this.utils = new ScraperUtils(this.config);
|
||||||
|
this.limit = limit;
|
||||||
|
}
|
||||||
|
|
||||||
|
formatSeenDate(dateString: string) {
|
||||||
|
const clean = dateString.split(" Uhr")[0];
|
||||||
|
|
||||||
|
const [datePart, timePart] = clean.split(" ");
|
||||||
|
const [day, month, year] = datePart.split(".");
|
||||||
|
|
||||||
|
const isoString = `${year}-${month}-${day}T${timePart}`;
|
||||||
|
return new Date(isoString);
|
||||||
|
}
|
||||||
|
|
||||||
|
async scrape(): Promise<void> {
|
||||||
|
const browser = await this.utils.launchBrowser();
|
||||||
|
|
||||||
|
try {
|
||||||
|
await this.utils.performLogin(browser);
|
||||||
|
const page = await this.utils.setupPage(browser);
|
||||||
|
await page.goto(this.config.searchUrl);
|
||||||
|
|
||||||
|
await this.scrapePages(page);
|
||||||
|
|
||||||
|
const totalMovies = Object.keys(this.movieTitles).length;
|
||||||
|
console.log(`\nFound ${totalMovies} unique series on aniworld.to`);
|
||||||
|
|
||||||
|
await this.utils.saveToFile(this.movieTitles, "aniworld.to");
|
||||||
|
} finally {
|
||||||
|
await browser.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async scrapePages(page: Page): Promise<void> {
|
||||||
|
let currentPage = 1;
|
||||||
|
const lastPageNum = await this.getLastPageNumber(page);
|
||||||
|
|
||||||
|
while (true) {
|
||||||
|
const hasMorePages = await this.scrapePage(
|
||||||
|
page,
|
||||||
|
currentPage,
|
||||||
|
lastPageNum,
|
||||||
|
);
|
||||||
|
|
||||||
|
if (!hasMorePages || (this.limit > 0 && currentPage >= this.limit)) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
currentPage++;
|
||||||
|
await page.goto(this.config.searchUrl + currentPage);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async scrapePage(
|
||||||
|
page: Page,
|
||||||
|
pageNum: number,
|
||||||
|
lastPageNum: number | null,
|
||||||
|
): Promise<boolean> {
|
||||||
|
try {
|
||||||
|
const sel = this.config.selectors;
|
||||||
|
const rowData = await page.$$eval(
|
||||||
|
sel.row,
|
||||||
|
(rows, s) =>
|
||||||
|
rows.map((r) => ({
|
||||||
|
entry: r.querySelector(s.title)?.textContent?.trim() ?? "",
|
||||||
|
info: r.querySelector(s.info)?.textContent?.trim() ?? "",
|
||||||
|
date: r.querySelector(s.time)?.textContent?.trim() ?? "",
|
||||||
|
})),
|
||||||
|
sel,
|
||||||
|
);
|
||||||
|
|
||||||
|
if (rowData.length === 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const data of rowData) {
|
||||||
|
this.processRow(data);
|
||||||
|
}
|
||||||
|
|
||||||
|
process.stdout.clearLine(0);
|
||||||
|
process.stdout.cursorTo(0);
|
||||||
|
process.stdout.write(`Progress: ${pageNum} / ${lastPageNum || "?"}`);
|
||||||
|
|
||||||
|
return pageNum < (lastPageNum || pageNum);
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Error scraping page ${pageNum}:`, error);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private processRow(data: { entry: string; info: string; date: string }): void {
|
||||||
|
try {
|
||||||
|
const title = data.entry.split(" - ")[0];
|
||||||
|
const [season, episode] = data.entry
|
||||||
|
.split(" - ")[1]
|
||||||
|
.split(" ")
|
||||||
|
.map((v: string) => parseInt(v.substring(1)));
|
||||||
|
|
||||||
|
const seasonNumber = season > 0 ? season.toString() : "Movie";
|
||||||
|
|
||||||
|
if (!this.movieTitles[title]) {
|
||||||
|
this.movieTitles[title] = {
|
||||||
|
seasonCount: 0,
|
||||||
|
totalEpisodesCount: 0,
|
||||||
|
seasons: {},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!this.movieTitles[title].seasons[seasonNumber]) {
|
||||||
|
this.movieTitles[title].seasons[seasonNumber] = {
|
||||||
|
episodeCount: 0,
|
||||||
|
episodes: {},
|
||||||
|
};
|
||||||
|
this.movieTitles[title].seasonCount++;
|
||||||
|
}
|
||||||
|
|
||||||
|
this.movieTitles[title].totalEpisodesCount++;
|
||||||
|
this.movieTitles[title].seasons[seasonNumber].episodeCount++;
|
||||||
|
this.movieTitles[title].seasons[seasonNumber].episodes[
|
||||||
|
episode.toString()
|
||||||
|
] = {
|
||||||
|
title: data.info,
|
||||||
|
seenAt: this.formatSeenDate(data.date),
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
console.error("Error processing row:", error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async getLastPageNumber(page: Page): Promise<number | null> {
|
||||||
|
try {
|
||||||
|
const lastPage = await page.$(
|
||||||
|
`${this.config.selectors.nav} li:last-child a`,
|
||||||
|
);
|
||||||
|
const lastPageNum = await lastPage?.evaluate((el) => {
|
||||||
|
const url = el.href;
|
||||||
|
return url.split("/").pop()?.replace("#", "");
|
||||||
|
});
|
||||||
|
|
||||||
|
if (lastPageNum && !isNaN(parseInt(lastPageNum))) {
|
||||||
|
return parseInt(lastPageNum);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
console.error("Error getting last page number:", error);
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// CLI execution
|
||||||
|
if (require.main === module) {
|
||||||
|
const limit = parseInt(process.argv[2]) || 0;
|
||||||
|
const scraper = new AniworldScraper(limit);
|
||||||
|
scraper.scrape().catch(console.error);
|
||||||
|
}
|
||||||
177
src/scrapers/scraper-sto.ts
Normal file
177
src/scrapers/scraper-sto.ts
Normal file
|
|
@ -0,0 +1,177 @@
|
||||||
|
import { Page } from "playwright";
|
||||||
|
import { SeriesData } from "../types";
|
||||||
|
import { ScraperUtils } from "./scraper-utils";
|
||||||
|
|
||||||
|
const STO_CONFIG = {
|
||||||
|
website: "serienstream.to",
|
||||||
|
loginUrl: "https://serienstream.to/login",
|
||||||
|
searchUrl: "https://serienstream.to/account/watch-history",
|
||||||
|
selectors: {
|
||||||
|
card: ".card",
|
||||||
|
title: "h2 a",
|
||||||
|
info: "p a",
|
||||||
|
time: "span.float-end",
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
export class StoScraper {
|
||||||
|
private utils: ScraperUtils;
|
||||||
|
private config: typeof STO_CONFIG;
|
||||||
|
private movieTitles: { [title: string]: SeriesData } = {};
|
||||||
|
private limit: number;
|
||||||
|
|
||||||
|
constructor(limit: number = 0) {
|
||||||
|
this.config = STO_CONFIG as typeof STO_CONFIG;
|
||||||
|
this.utils = new ScraperUtils(this.config);
|
||||||
|
this.limit = limit;
|
||||||
|
}
|
||||||
|
|
||||||
|
formatSeenDate(dateString: string) {
|
||||||
|
const clean = dateString.split(" Uhr")[0];
|
||||||
|
|
||||||
|
const [datePart, timePart] = clean.split(" ");
|
||||||
|
const [year, month, day] = datePart.split("-");
|
||||||
|
|
||||||
|
const isoString = `${year}-${month}-${day}T${timePart}`;
|
||||||
|
return new Date(isoString);
|
||||||
|
}
|
||||||
|
|
||||||
|
async scrape(): Promise<void> {
|
||||||
|
const browser = await this.utils.launchBrowser();
|
||||||
|
|
||||||
|
try {
|
||||||
|
await this.utils.performLogin(browser);
|
||||||
|
const page = await this.utils.setupPage(browser);
|
||||||
|
await page.goto(this.config.searchUrl);
|
||||||
|
|
||||||
|
await this.scrapePages(page);
|
||||||
|
|
||||||
|
const totalMovies = Object.keys(this.movieTitles).length;
|
||||||
|
console.log(`\nFound ${totalMovies} unique series on serienstream.to`);
|
||||||
|
|
||||||
|
await this.utils.saveToFile(this.movieTitles, "serienstream.to");
|
||||||
|
} finally {
|
||||||
|
await browser.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async scrapePages(page: Page): Promise<void> {
|
||||||
|
let currentPage = 1;
|
||||||
|
let cursor: string | null = null;
|
||||||
|
|
||||||
|
while (true) {
|
||||||
|
const { hasMorePages, nextCursor } = await this.scrapePage(
|
||||||
|
page,
|
||||||
|
currentPage,
|
||||||
|
cursor,
|
||||||
|
);
|
||||||
|
|
||||||
|
if (!hasMorePages || (this.limit > 0 && currentPage >= this.limit)) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
cursor = nextCursor;
|
||||||
|
currentPage++;
|
||||||
|
await page.goto(this.config.searchUrl + `?cursor=` + cursor);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async scrapePage(
|
||||||
|
page: Page,
|
||||||
|
pageNum: number,
|
||||||
|
cursor: string | null,
|
||||||
|
): Promise<{ hasMorePages: boolean; nextCursor: string | null }> {
|
||||||
|
try {
|
||||||
|
const sel = this.config.selectors;
|
||||||
|
const cardData = await page.$$eval(
|
||||||
|
sel.card,
|
||||||
|
(cards, s) =>
|
||||||
|
cards.map((c) => ({
|
||||||
|
title: c.querySelector(s.title)?.textContent?.trim() ?? "",
|
||||||
|
info: (c.querySelector(s.info) as HTMLElement)?.innerText?.trim() ?? "",
|
||||||
|
date: c.querySelector(s.time)?.textContent?.trim() ?? "",
|
||||||
|
})),
|
||||||
|
sel,
|
||||||
|
);
|
||||||
|
|
||||||
|
if (cardData.length === 0) {
|
||||||
|
return {
|
||||||
|
hasMorePages: false,
|
||||||
|
nextCursor: null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const data of cardData) {
|
||||||
|
this.processCard(data);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (process.stdout.isTTY) {
|
||||||
|
process.stdout.clearLine(0);
|
||||||
|
process.stdout.cursorTo(0);
|
||||||
|
process.stdout.write(`Progress: ${pageNum} / ?`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const nextPage = await page.$eval(
|
||||||
|
"ul.pagination :nth-child(2) a",
|
||||||
|
(el: Element) => {
|
||||||
|
const isDisabled = el.classList.contains("disabled");
|
||||||
|
const nextUrl = new URL(el.getAttribute("href")!);
|
||||||
|
const cursor = nextUrl.searchParams.get("cursor");
|
||||||
|
return { hasMorePages: !isDisabled, nextCursor: cursor || null };
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
return {
|
||||||
|
hasMorePages: nextPage.hasMorePages,
|
||||||
|
nextCursor: nextPage.nextCursor,
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Error scraping page ${pageNum}:`, error);
|
||||||
|
return {
|
||||||
|
hasMorePages: false,
|
||||||
|
nextCursor: null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private processCard(data: { title: string; info: string; date: string }): void {
|
||||||
|
try {
|
||||||
|
const [seasonEpisode, episodeTitle] = data.info.split(" – ");
|
||||||
|
const [seasonNumber, episodeNumber] = seasonEpisode
|
||||||
|
.split(" ")
|
||||||
|
.map((v: string) => parseInt(v.substring(1)) || 0);
|
||||||
|
|
||||||
|
if (!this.movieTitles[data.title]) {
|
||||||
|
this.movieTitles[data.title] = {
|
||||||
|
seasonCount: 0,
|
||||||
|
totalEpisodesCount: 0,
|
||||||
|
seasons: {},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!this.movieTitles[data.title].seasons[seasonNumber]) {
|
||||||
|
this.movieTitles[data.title].seasons[seasonNumber] = {
|
||||||
|
episodeCount: 0,
|
||||||
|
episodes: {},
|
||||||
|
};
|
||||||
|
this.movieTitles[data.title].seasonCount++;
|
||||||
|
}
|
||||||
|
|
||||||
|
this.movieTitles[data.title].totalEpisodesCount++;
|
||||||
|
this.movieTitles[data.title].seasons[seasonNumber].episodeCount++;
|
||||||
|
this.movieTitles[data.title].seasons[seasonNumber].episodes[episodeNumber] = {
|
||||||
|
title: episodeTitle,
|
||||||
|
seenAt: this.formatSeenDate(data.date),
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
console.error("Error processing card:", error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// CLI execution
|
||||||
|
if (require.main === module) {
|
||||||
|
const limit = parseInt(process.argv[2]) || 0;
|
||||||
|
const scraper = new StoScraper(limit);
|
||||||
|
scraper.scrape().catch(console.error);
|
||||||
|
}
|
||||||
145
src/scrapers/scraper-utils.ts
Normal file
145
src/scrapers/scraper-utils.ts
Normal file
|
|
@ -0,0 +1,145 @@
|
||||||
|
import { firefox, Browser, Page } from "playwright";
|
||||||
|
import dotenv from "dotenv";
|
||||||
|
import fs from "fs";
|
||||||
|
import path from "path";
|
||||||
|
import { ScraperConfig, ScraperOutput, SeriesData } from "../types";
|
||||||
|
|
||||||
|
dotenv.config();
|
||||||
|
|
||||||
|
const FIREFOX_UA =
|
||||||
|
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) Gecko/20100101 Firefox/140.0";
|
||||||
|
|
||||||
|
export class ScraperUtils {
|
||||||
|
private config: ScraperConfig;
|
||||||
|
private headless: boolean;
|
||||||
|
private dontLoadStyles: boolean;
|
||||||
|
private authenticatedPage: Page | null = null;
|
||||||
|
|
||||||
|
constructor(config: ScraperConfig) {
|
||||||
|
this.config = config;
|
||||||
|
this.headless = process.env.HEADLESS === "true";
|
||||||
|
this.dontLoadStyles = process.env.DONT_LOAD_STYLES === "true";
|
||||||
|
}
|
||||||
|
|
||||||
|
async launchBrowser(): Promise<Browser> {
|
||||||
|
return await firefox.launch({ headless: this.headless });
|
||||||
|
}
|
||||||
|
|
||||||
|
async fillCredentials(page: Page): Promise<void> {
|
||||||
|
await page.waitForSelector('input[name="email"]');
|
||||||
|
await page.fill('input[name="email"]', process.env.PAGE_USERNAME || "");
|
||||||
|
await page.fill('input[name="password"]', process.env.PAGE_PASSWORD || "");
|
||||||
|
}
|
||||||
|
|
||||||
|
async performLogin(browser: Browser): Promise<void> {
|
||||||
|
const loginPage = await browser.newPage({ userAgent: FIREFOX_UA });
|
||||||
|
await loginPage.goto(this.config.loginUrl, { waitUntil: "domcontentloaded" });
|
||||||
|
|
||||||
|
console.log("Logging in...");
|
||||||
|
|
||||||
|
await this.fillCredentials(loginPage);
|
||||||
|
await loginPage.keyboard.press("Enter");
|
||||||
|
try {
|
||||||
|
await loginPage.waitForURL("**/account**", { timeout: 30000 });
|
||||||
|
} catch {
|
||||||
|
// navigation did not land on an /account page
|
||||||
|
}
|
||||||
|
|
||||||
|
const url = loginPage.url();
|
||||||
|
const loggedIn = url.includes("/account");
|
||||||
|
|
||||||
|
if (!loggedIn) {
|
||||||
|
console.log("Login failed.");
|
||||||
|
const captcha = await loginPage.$(".messageAlert.danger");
|
||||||
|
if (captcha) {
|
||||||
|
if (this.headless) {
|
||||||
|
console.log("Captcha detected, please set headless to false in .env");
|
||||||
|
process.exit(1);
|
||||||
|
} else {
|
||||||
|
console.log("Captcha detected, please solve it manually");
|
||||||
|
await this.fillCredentials(loginPage);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
console.log("Logged in successfully");
|
||||||
|
this.authenticatedPage = loginPage;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async setupPage(browser: Browser): Promise<Page> {
|
||||||
|
const page = this.authenticatedPage ?? (await browser.newPage({ userAgent: FIREFOX_UA }));
|
||||||
|
|
||||||
|
if (this.dontLoadStyles) {
|
||||||
|
await page.route("**/*", (route) => {
|
||||||
|
const type = route.request().resourceType();
|
||||||
|
if (
|
||||||
|
type === "stylesheet" ||
|
||||||
|
type === "font" ||
|
||||||
|
type === "image"
|
||||||
|
) {
|
||||||
|
route.abort();
|
||||||
|
} else {
|
||||||
|
route.continue();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return page;
|
||||||
|
}
|
||||||
|
|
||||||
|
getGlobalEpisodeCount(data: ScraperOutput): number {
|
||||||
|
return Object.values(data.movieTitles)
|
||||||
|
.flatMap((series) => Object.values(series.seasons))
|
||||||
|
.reduce((sum, season) => {
|
||||||
|
const count = season.episodes ? Object.keys(season.episodes).length : 0;
|
||||||
|
|
||||||
|
return sum + count;
|
||||||
|
}, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
async saveToFile(movieTitles: { [p: string]: SeriesData }, site: string): Promise<void> {
|
||||||
|
const dataDir = "./data/";
|
||||||
|
const data: ScraperOutput = {} as ScraperOutput;
|
||||||
|
|
||||||
|
if (!fs.existsSync(dataDir)) {
|
||||||
|
fs.mkdirSync(dataDir, { recursive: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
const d = new Date();
|
||||||
|
const datestring =
|
||||||
|
d.getFullYear() +
|
||||||
|
"-" +
|
||||||
|
("0" + (d.getMonth() + 1)).slice(-2) +
|
||||||
|
"-" +
|
||||||
|
("0" + d.getDate()).slice(-2) +
|
||||||
|
"_" +
|
||||||
|
("0" + d.getHours()).slice(-2) +
|
||||||
|
"-" +
|
||||||
|
("0" + d.getMinutes()).slice(-2) +
|
||||||
|
"-" +
|
||||||
|
("0" + d.getSeconds()).slice(-2);
|
||||||
|
|
||||||
|
// Versioned file
|
||||||
|
const versionedPath = path.join(dataDir, `${site}_${datestring}.json`);
|
||||||
|
|
||||||
|
// Latest file (always overwritten)
|
||||||
|
const latestPath = path.join(dataDir, `${site}_latest.json`);
|
||||||
|
|
||||||
|
// Add metadata
|
||||||
|
data.movieTitles = movieTitles;
|
||||||
|
data.scrapedAt = new Date().toISOString();
|
||||||
|
data.site = site;
|
||||||
|
data.totalMovies = Object.keys(movieTitles).length;
|
||||||
|
data.totalEpisodesCount = this.getGlobalEpisodeCount(data);
|
||||||
|
|
||||||
|
const json = JSON.stringify(data, null, 2);
|
||||||
|
|
||||||
|
await Promise.all([
|
||||||
|
fs.promises.writeFile(versionedPath, json),
|
||||||
|
fs.promises.writeFile(latestPath, json),
|
||||||
|
]);
|
||||||
|
|
||||||
|
console.log(`\nData saved to ${versionedPath}`);
|
||||||
|
console.log(`Latest updated: ${latestPath}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
98
src/scripts/list-series.ts
Normal file
98
src/scripts/list-series.ts
Normal file
|
|
@ -0,0 +1,98 @@
|
||||||
|
import fs from "fs";
|
||||||
|
import path from "path";
|
||||||
|
import { ScraperOutput } from "../types";
|
||||||
|
|
||||||
|
function findLatestDataFile(): string | null {
|
||||||
|
const dataDir = path.join(process.cwd(), "data");
|
||||||
|
|
||||||
|
if (!fs.existsSync(dataDir)) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const files = fs
|
||||||
|
.readdirSync(dataDir)
|
||||||
|
.filter((file) => file.endsWith(".json"));
|
||||||
|
|
||||||
|
if (files.length === 0) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
return files
|
||||||
|
.map((file) => ({ file: path.join(dataDir, file), time: fs.statSync(path.join(dataDir, file)).mtimeMs }))
|
||||||
|
.sort((a, b) => b.time - a.time)[0]!.file;
|
||||||
|
}
|
||||||
|
|
||||||
|
function printHelp(): void {
|
||||||
|
console.log("Usage: npm run list:series [path-to-data.json]");
|
||||||
|
console.log(" With no path, the newest *.json file in ./data is used.");
|
||||||
|
}
|
||||||
|
|
||||||
|
function main(): void {
|
||||||
|
const arg = process.argv[2];
|
||||||
|
const filePath = arg
|
||||||
|
? path.resolve(process.cwd(), arg)
|
||||||
|
: findLatestDataFile();
|
||||||
|
|
||||||
|
if (!filePath) {
|
||||||
|
console.error("No data file found. Run the scraper first (npm run scrape:sto).");
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!fs.existsSync(filePath)) {
|
||||||
|
console.error(`Data file does not exist: ${filePath}`);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
const data = JSON.parse(fs.readFileSync(filePath, "utf-8")) as ScraperOutput;
|
||||||
|
const entries = Object.entries(data.movieTitles);
|
||||||
|
|
||||||
|
const totalEpisodes = entries.reduce(
|
||||||
|
(sum, [, series]) => sum + series.totalEpisodesCount,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
|
||||||
|
const header = [
|
||||||
|
"Series Watch List",
|
||||||
|
"=================",
|
||||||
|
`Source: ${path.relative(process.cwd(), filePath)}`,
|
||||||
|
`Site: ${data.site}`,
|
||||||
|
`Scraped at: ${data.scrapedAt}`,
|
||||||
|
"",
|
||||||
|
`Total series: ${entries.length}`,
|
||||||
|
`Watched episodes: ${totalEpisodes}`,
|
||||||
|
"",
|
||||||
|
].join("\n");
|
||||||
|
|
||||||
|
const rows = entries
|
||||||
|
.map(([title, series]) => ({
|
||||||
|
title,
|
||||||
|
episodes: series.totalEpisodesCount,
|
||||||
|
seasons: series.seasonCount,
|
||||||
|
}))
|
||||||
|
.sort((a, b) => b.episodes - a.episodes || a.title.localeCompare(b.title));
|
||||||
|
|
||||||
|
const line = rows
|
||||||
|
.map(
|
||||||
|
(row) =>
|
||||||
|
`${String(row.episodes).padStart(5)} eps ` +
|
||||||
|
`${String(row.seasons).padStart(3)} season(s) ${row.title}`,
|
||||||
|
)
|
||||||
|
.join("\n");
|
||||||
|
|
||||||
|
const footer =
|
||||||
|
"\n" +
|
||||||
|
"─".repeat(50) +
|
||||||
|
`\n${entries.length} series, ${totalEpisodes} watched episodes total.`;
|
||||||
|
|
||||||
|
console.log(`${header}\n${line}${footer}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (require.main === module) {
|
||||||
|
if (process.argv[2] === "-h" || process.argv[2] === "--help") {
|
||||||
|
printHelp();
|
||||||
|
} else {
|
||||||
|
main();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export { main };
|
||||||
40
src/types.ts
Normal file
40
src/types.ts
Normal file
|
|
@ -0,0 +1,40 @@
|
||||||
|
// Shared types for the scraper project
|
||||||
|
|
||||||
|
export interface EpisodeData {
|
||||||
|
title: string;
|
||||||
|
seenAt: Date;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SeasonData {
|
||||||
|
episodeCount: number;
|
||||||
|
episodes: {
|
||||||
|
[episodeNumber: string]: EpisodeData;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SeriesData {
|
||||||
|
seasonCount: number;
|
||||||
|
totalEpisodesCount: number;
|
||||||
|
seasons: {
|
||||||
|
[seasonNumber: string]: SeasonData;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ScraperOutput {
|
||||||
|
scrapedAt: string;
|
||||||
|
site: string;
|
||||||
|
totalMovies: number;
|
||||||
|
totalEpisodesCount: number;
|
||||||
|
movieTitles: {
|
||||||
|
[title: string]: SeriesData;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ScraperConfig {
|
||||||
|
website: string;
|
||||||
|
loginUrl: string;
|
||||||
|
searchUrl: string;
|
||||||
|
selectors: {
|
||||||
|
[key: string]: string;
|
||||||
|
};
|
||||||
|
}
|
||||||
20
tsconfig.json
Normal file
20
tsconfig.json
Normal file
|
|
@ -0,0 +1,20 @@
|
||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2020",
|
||||||
|
"module": "commonjs",
|
||||||
|
"lib": ["ES2020", "DOM"],
|
||||||
|
"outDir": "./dist",
|
||||||
|
"rootDir": "./src",
|
||||||
|
"strict": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"resolveJsonModule": true,
|
||||||
|
"moduleResolution": "node",
|
||||||
|
"ignoreDeprecations": "6.0",
|
||||||
|
"typeRoots": ["node_modules/@types"],
|
||||||
|
"types": ["node"]
|
||||||
|
},
|
||||||
|
"include": ["src/**/*"],
|
||||||
|
"exclude": ["node_modules", "dist"]
|
||||||
|
}
|
||||||
Loading…
Reference in a new issue