SQL Query Looking for optimisation for complex query with indexes

$query = "SELECT * 
          FROM $database1 
          WHERE userid!='$userid' 
            AND mediaid NOT IN (SELECT mediaid 
                                FROM $database2 
                                WHERE uid='$userid')
          ORDER BY active ASC LIMIT 80";

This query was working very well until now.

That it suddenly takes 0.5 to sometimes even 3s to execute, sometimes it also goes down to 0.1 which is acceptable.

Now database2 has around 3 millionn rows and database1 around 500, but it also sometimes is slow when only 100 items in database1. I'm worried because database2 is getting around 30k new rows each day.

Both mediaid, userid, uid are all indexes.

Server? 8cores x 3,2, 16gb ram. Scalable cloud. Average load is fine. not more than 20% cpus