Memewidth

So, we know that when LJ is on the fritz it means a spike in DW activity until LJ smooths out and most people go back. You can see some pretty serious spikes on the newbyday stat:

newbyday 2011-04-02 402
newbyday 2011-04-03 243
newbyday 2011-04-04 2506
newbyday 2011-04-05 3658
newbyday 2011-04-06 16226
newbyday 2011-04-07 22136
newbyday 2011-04-08 10839
newbyday 2011-04-09 9735
newbyday 2011-04-10 4214
newbyday 2011-04-11 468
newbyday 2011-04-12 6897
newbyday 2011-04-13 8317

Of course, newbyday includes ALL accounts, including OpenID accounts that are created during imports through comments or access giving. That's why 4/11 is so low compared to the others--the importer had been shut down.

So in order to gauge activity spikes, it's easier to look at other metrics like active/posting, which would include people coming back to fallow accounts they haven't touched in a month as well as new accounts that have been signing in and posting content. Here's the numbers comparing 3/31/2011 to 4/13/2011 (just after 12AM two weeks apart to the day, so it's a better comparison than different days, since activity varies through the week):

Stat	3/31/2011	4/13/2011	# increase	% increase
Active	20,616	29,685	9,069	44%
Posted last 30 days	18,448	22,633	4,185	22.6%
Posted last 7 days	12,341	16,311	3,970	33.2%
Posted last 24 hours	5,690	6,847	1,157	20.3%

So, this is my offering to three_weeks_for_dw: I am making graphs of participant circles for posts and comments to communities! (And I hope to continue making more kinds of fun graphs you can get by certain activities.) They can be explained thusly:

• The graph consists of nodes (the circles with the journals names on them) and edges (the lines between the nodes).
• A line between two nodes means that those two users have a mutual relationship. That means they subscribe to each other, give each other access, or both.
• The size of the node indicates how many mutual relationships that user has in total--the more, the bigger the node.
• The color of the node indicates how many relative mutual relationships that user has with you--the darker the red, the more connections you two share.
• Note that you don't show up in the graphs, since you are connected to everybody
• If you have a relationship with a user but they don't have any relationships with anybody else you're connected to, they don't show up either (I call them "singlets").

( Example time! )

You can get a network graph if the account you're requesting if with that account you have made, since April 26th or later:

• Three posts to three communities.
• Comments to six different posts in at least three different communities.

For each additional post or 3 comments you make to a community, you can choose one of the bonus add ons:

• A list of all of the singlets--people who don't have any mutual relationships with other people in your circle
• A graph that highlights connections between people in your biggest clique and a list of your biggest clique(s)
• For you geeks, a dot file you can play around with in GraphViz.

Put the links to the posts or comments in your request, and follow the rules of the communities you are posting to!
Unfortunately, the posts and comments will need to be public ones, so I can see them and verify. By default, I will put your graph(s) up on my web hosting and link you to it, so you can point to it in your journal easily (although it really is best if you host it for yourself, just in case), but if you'd prefer me to email it to you, just let me know.

If you don't know how to find communities you're interested in, feel free to make a comment here asking me for possible recommendations. I or others passing by will try to find some for you--though I'll note that I'm not in fandom, so my experience will be limited in that area.

Every now and again I get the urge to do something with mass amounts of post data I'm accumulating, and I thought I'd share a useful tool for text analysis, which is lists of words:

http://wordlist.sourceforge.net/

I'm currently considering using the 2+2lemma.txt from 12dicts for something, to reduce the number of distinct words. I'll probably have to add common slang words, though, and names for services like Twitter/Facebook/MySpace/GMail/etc.

ETA: Social Media, Data Mining & Machine Learning might be an interesting blog read along these lines. And if you have University access, you may be able to get ahold of Modeling and Data Mining
in Blogosphere.

* Now, the username and account number are available, as "name" and "account_id" values.
* OpenID accounts now use their username forms (ext_12345) to simplify spidering (some OpenID URLs are tricky to fetch as a URL parameter), and have "display_name" values for their edges.

Edge data is live from last night's code push, folks! In celebration, I've made wiki pages on some Dreamwidth data sources:

http://wiki.dwscoalition.org/notes/Data_Sources

After some further refinements by mark, edge patches have been committed! That means they're going out in the next code push! Exciting!

You can see the format here (I recommend JSONView) for viewing):

http://www.dev.memewidth.org/data/edges?user=amy

Names have been replaced with userids, which are then defined with username/type in a different struct.

Now that fdata.bml is in JSON, would it be expedient to make interests.bml into JSON?

Pros: would be easier for Javascript mashups to load, is a standard format for parsing
Cons: would break compatibility with older LJ-based tools, would take up a bit more space than the current file.

So far Dreamwidth has been sorely lacking an equivalent to LJ's fdata, the data file that told you the relationships of a user with other users. Our new equivalent to that will be called edges. Because of the new complexity of DW's relationship structure, and due to a desire to use more standards, this data file will be in the JSON format. The bug for it is Bug 857.

I currently have my proposed patch in. It's not committed yet, or live on the site, but I think it probably will be within the next month. If you want a chance to prepare, I have my patch applied on my development server. It's possible that details will change with feedback, though, so be forewarned! My server has open registration, so you can feel free to create accounts and play around, but I've made a small set of test accounts such as this one. You can view an example edge data file here:

http://www.dev.memewidth.org/data/edges?account=amy

Here is an example of a community's edges file:

http://www.dev.memewidth.org/data/edges?account=bonny_bunnies

The data is in one big hash. There is an "account_type" variable, set to the type of the account. That makes it easy to determine what kind of account you are fetching data for. The other base variables are relationship types, such as "watched", "watched_by", or "member_of", that are a hash of arrays--all the accounts with a given relationship are split up into account types.

Any commentary on this planned data file? Problems you want to fix now, before it's implemented? Any plans for it you want to share?

Hello, memers! Boy, there sure are a lot more of you than there were two weeks ago, because of how much DW is growing. With the help of some scripts I set up a while ago and sophie's charting prowess, ( we've made a graph of exactly how much DW has grown lately. )