Simple console spam detector for YouTube chats.
Monitors a set of YouTube channels, and starts collecting messages as soon as a new chat room opens. All collected messages are then sent to an embedded spam detector and also saved to a database for a further analysis when searching for a ways to improve spam detection.
Consists of several modules:
- core - common types and objects used by other modules
- StreamFinder - monitors YouTube channels for live and upcoming streams and premiers
- ChatManager - collects messages from every open chat room
- Detector - analyses messages and tries to detect potential spammers
- CLI - temporary CLI front end
DB- saves all messages and desicions, made byDetectorto a database (module is not implemented yet)WEB- web front end, that allows to add and remove channels from monitoring and to change various aspects of spam detection (module is not implemented yet)
All modules, except core are implemented as independend actors, which should allow for an easy horizontal scallability in the future, if such a need ever arises.
This app doesn't use YouTube API, and instead tries to emulate the behaviour of a browser. The reason for this decision is that YouTube by default provides only 10 thousand credits a day to spend on requests. 1 request to load new messages from the chat costs 5 credits, and, depending on how active the chat, should be performed every 5-10 seconds. If we assume that on average the app would perform 5 requests per minute, we can estimate that we will spend around 1500 credits per hour.
And some channels either stream for 24/7, for example Lofi Girl, or have streams planned far into the future, that effectively act as a chat rooms for viewers without the need to create Discord server. That's ~36000 credits per day for each such stream/chat room.
Moreover, to get the list of live broadcasts, we would have to use Search API, which costs 100 credist for each request, meaning that we can only make 1 request every ~15 minutes. And then we would have only 400 credits left to actually collect chat messages. And that's only for 1 channel.
At the moment, all parameters are hardcoded inside the main.rs file. Spam detection parameters are stored inside DetectorParams struct.
The list of channels to track is stored inside channels hash set (lines 73-85). The app is using channel id when adding a new channel to track, but some YouTube channels use custom user name instead of channel id (e.g. https://www.youtube.com/user/PewDiePie). In that case you need to open any video from that channel, and then click on channel's name under the video. This would open the same channel page, but this time instead of a custom user name, you'll see a channel id in the browser's address bar (e.g. https://www.youtube.com/channel/UC-lHJZR3Gqxm24_Vd_AJ5Yw for PewDiePie).
To actually run the app, just run cargo run --release