The domain https://fediverse.observer/ redirects to sedo now.
On October 11th these two commits where made to the repo:
this will be my final release, glad to have contributed since 2011 to some fediverse stats, https://fedidb.com/ is a great resource with more interest
this software is no longer developed
Sad to have lost such a great resource.
Yeah. I saw the dead website earlier today and have been trying to figure out how to keep the posts working.
The fediverse software post, I could rewrite some code to gather fedidb daily (to my knowledge, they don’t have all time series data available), but their MAU math is suspect. So, not a great option, but my only other thought is basically remaking fediverse-observer and I do not have a server or potentially relevant server skill set beyond cron jobs.
For the threadiverse instances post, I can keep gathering data from the instances I already have in my database, but don’t really have another way to find new ones. Maybe, I could see if I could scrape it from fedidb (I couldn’t get their API working in the past).
In conclusion, the posts are not in a good place :(
Man that sucks. But also because I never have before, I want to thank you for all your labor making those posts! I find them very interesting.
Glad you enjoy them :)
Just need to figure out how to keep them going properly. I have a couple thoughts I’ll look into tomorrow. Hopefully one of them bears fruit.
Good luck!
Do they do any math? I assumed it’s simply additive. They get the MAU from individual instances via their crawler, then display a total?
Differences to other sites is probably due to the set of instances checked, not anything math related.
https://fedidb.com/software/lemmy?search=lemmy.world&stats=1
Fedidb doesn’t seem to include lemmy.world, which seems like a substantial omission, as I believe that it’s the largest Lemmy instance.
EDIT: And my own home instance, lemmy.today, also doesn’t seem to be on there:
https://fedidb.com/software/lemmy?search=lemmy.today
The largest user count server that they seem to query is lemmy.ml:
https://fedidb.com/software/lemmy?search=lemmy.ml
EDIT2: Your home instance, kbin.earth, does seem to be on there:
https://fedidb.com/servers/kbin.earth
That is interesting to know. It doesn’t look like lemmy.world has opted out, judging by their robots.txt.
Oh, that’s a thought. I didn’t think of checking robots.txt.
Lemmy.today does have a blanket ban on all bots in its robots.txt, which I assume was put into place to help try to reduce (sporadically severe) scraping load. I’m not sure that the admin, @mrmanager@lemmy.today, realized that it might have cut it out of Lemmy statistics gathering.
Yeah I know, but its worth it. :)
I knew to check robots.txt because of fedidb’s Data Sources page, which specifies what data they get, their user agent, and how to opt-out via robots.txt.
One thing I’m noticing in regards to lemmy.world, I wonder if they ever disallowed access to the nodeinfo endpoint in their robots.txt in the past. They don’t currently, but the wording suggests that having that rule at any point will forever remove the instance from fedidb. Which could explain what we’re seeing for that instance.
You’d think it’s straightforward and additive. I added up all their listed instances for a couple different softwares. The math was wrong and overestimated the MAU count. I checked it a few times across a couple weeks. Couldn’t quickly figure out a pattern to how the math was wrong each time.
Maybe they fixed it in the last ~6 months, but I wouldn’t count on it. I believe it’s the same Dev as pixelfed and loops. So, not sure they’ve done any work on fedidb in years (to my knowledge, they’ve focused on Loops and Pixelfed).
Edit: to clarify, adding the MAU of all their listed instances for a particular software was lower than the total MAU they reported for the software.
Edit 2: Just checked their Piefed listings and it’s still wrong. Fedidb states: ~7.0k, actual addition: 6858 MAU.
Have you contacted them about the discrepancy? Ideally FediDB could just become the new Fediverse Observer.
Missing out on accurate data about Fediverse growth would really suck. 😕
I just confirmed the problem still exists and messaged them via mastodon about it.
Probably didn’t let them know originally because I didn’t have a mastodon account at the time and already had a good option (fediverse-observer).