Memewidth

six-degrees-esque tool

Mon, 25 Apr 2011 05:27:43 GMT

Posted by: exor674

http://edges.andreanall.com/ -- Enter a username in box A, enter a username in box B and the site will give you a path between the two users. For this to work, both accounts have to had existed since mid-March. You can choose for the tool to consider "Mutual Trust", "Mutual Watch", "Mutual Trust ( with membership )", "Any Edge", and "Using Interests". Please note that "Any Edge" and "Using Interests" have quite strict limits on how many accounts will be considered.

If anyone is interested in the underlying data, feel free to comment and I'll give you a link ( note however that it'll be something that's not in a format you can easily read or in a format that is likely to cause massive issues when unpacked on pretty much any filesystem )

comments

Recent DW stats

Fri, 15 Apr 2011 00:39:02 GMT

Posted by: foxfirefey

So, we know that when LJ is on the fritz it means a spike in DW activity until LJ smooths out and most people go back. You can see some pretty serious spikes on the newbyday stat:

newbyday 2011-04-02 402
newbyday 2011-04-03 243
newbyday 2011-04-04 2506
newbyday 2011-04-05 3658
newbyday 2011-04-06 16226
newbyday 2011-04-07 22136
newbyday 2011-04-08 10839
newbyday 2011-04-09 9735
newbyday 2011-04-10 4214
newbyday 2011-04-11 468
newbyday 2011-04-12 6897
newbyday 2011-04-13 8317

Of course, newbyday includes ALL accounts, including OpenID accounts that are created during imports through comments or access giving. That's why 4/11 is so low compared to the others--the importer had been shut down.

So in order to gauge activity spikes, it's easier to look at other metrics like active/posting, which would include people coming back to fallow accounts they haven't touched in a month as well as new accounts that have been signing in and posting content. Here's the numbers comparing 3/31/2011 to 4/13/2011 (just after 12AM two weeks apart to the day, so it's a better comparison than different days, since activity varies through the week):

Stat	3/31/2011	4/13/2011	# increase	% increase
Active	20,616	29,685	9,069	44%
Posted last 30 days	18,448	22,633	4,185	22.6%
Posted last 7 days	12,341	16,311	3,970	33.2%
Posted last 24 hours	5,690	6,847	1,157	20.3%

comments

Danah Boyd on data collection

Wed, 05 May 2010 00:08:35 GMT

Posted by: foxfirefey

A critique of Facebook's approach to privacy: Privacy and publicity in the context of "Big Data"

This article has some very good considerations when it comes to doing data analysis on Big Data.

comments

User to user mutual relationship graphs

Fri, 30 Apr 2010 10:31:59 GMT

Posted by: foxfirefey

So, this is my offering to three_weeks_for_dw: I am making graphs of participant circles for posts and comments to communities! (And I hope to continue making more kinds of fun graphs you can get by certain activities.) They can be explained thusly:

The graph consists of nodes (the circles with the journals names on them) and edges (the lines between the nodes).
A line between two nodes means that those two users have a mutual relationship. That means they subscribe to each other, give each other access, or both.
The size of the node indicates how many mutual relationships that user has in total--the more, the bigger the node.
The color of the node indicates how many relative mutual relationships that user has with you--the darker the red, the more connections you two share.
Note that you don't show up in the graphs, since you are connected to everybody
If you have a relationship with a user but they don't have any relationships with anybody else you're connected to, they don't show up either (I call them "singlets").

( Example time! )

You can get a network graph if the account you're requesting if with that account you have made, since April 26th or later:

Three posts to three communities.
Comments to six different posts in at least three different communities.

For each additional post or 3 comments you make to a community, you can choose one of the bonus add ons:

A list of all of the singlets--people who don't have any mutual relationships with other people in your circle
A graph that highlights connections between people in your biggest clique and a list of your biggest clique(s)
For you geeks, a dot file you can play around with in GraphViz.

Put the links to the posts or comments in your request, and follow the rules of the communities you are posting to!
Unfortunately, the posts and comments will need to be public ones, so I can see them and verify. By default, I will put your graph(s) up on my web hosting and link you to it, so you can point to it in your journal easily (although it really is best if you host it for yourself, just in case), but if you'd prefer me to email it to you, just let me know.

If you don't know how to find communities you're interested in, feel free to make a comment here asking me for possible recommendations. I or others passing by will try to find some for you--though I'll note that I'm not in fandom, so my experience will be limited in that area.

comments

Word frequencies in posts

Sat, 20 Mar 2010 20:42:38 GMT

Posted by: foxfirefey

The following's a list of word frequencies from the latest page, which displays a truncated/cut version of the latest public posts to Dreamwidth. A word only counts once per post; this data wasn't cleaned for unicode entities, although I should do that, so it's not as perfect as it should be, but I thought some people might have a passing curiosity like I did. The following table only shows words that show up 500 times or more in posts between the beginning of the year and yesterday. All the words have been lowercased.

( Table of word frequency doom! )

comments

Useful analysis data: word lists

Wed, 24 Feb 2010 01:46:24 GMT

Posted by: foxfirefey

Every now and again I get the urge to do something with mass amounts of post data I'm accumulating, and I thought I'd share a useful tool for text analysis, which is lists of words:

http://wordlist.sourceforge.net/

I'm currently considering using the 2+2lemma.txt from 12dicts for something, to reduce the number of distinct words. I'll probably have to add common slang words, though, and names for services like Twitter/Facebook/MySpace/GMail/etc.

ETA: Social Media, Data Mining & Machine Learning might be an interesting blog read along these lines. And if you have University access, you may be able to get ahold of Modeling and Data Mining
in Blogosphere.

comments

Update to DWminion

Fri, 08 Jan 2010 20:57:59 GMT

Posted by: foxfirefey

It pretty much only spiders edges files still, but I fixed it now that they're only available at username.dreamwidth.org/data/edges

You can find the Mercurial library to my Python library in progress here: http://bitbucket.org/foxfirefey/dwminion/

comments

Some changes made to the edges file

Tue, 27 Oct 2009 14:17:51 GMT

Posted by: foxfirefey

* Now, the username and account number are available, as "name" and "account_id" values.
* OpenID accounts now use their username forms (ext_12345) to simplify spidering (some OpenID URLs are tricky to fetch as a URL parameter), and have "display_name" values for their edges.

comments

Stats on popular entry properties

Mon, 19 Oct 2009 21:08:26 GMT

Posted by: foxfirefey

mark made a post on stats on popular entry properties, aka logprops. They'll be used to figure out how to organize a UI update of the entry page!

comments

S2 Client Protocol

Wed, 16 Sep 2009 00:12:47 GMT

Posted by: lastdance

If this isn't the right place, I'd be happy to be directed elsewhere. I'm looking for any documentation or examples around Dreamwidth's S2 style interface.

I know LJ has a few pages about it in their S2 manual. I can see that http://www.dreamwidth.org/interface/s2/layerid correctly retrives S2 layers, so it's available on Dreamwidth as well.

Are there any existing clients that use this interface for retrieving/uploading layers? Does Dreamwidth's implementation differ from LJ (for example, is it still required to send "application/x-danga-s2-layer" as Content-Type)?

I'd like to be able to use this interface to dynamically change my style layer, so I thought I'd check if any guidelines/best practices/documentation existed before I resorted to the trial-and-error method.

comments

Styles stats

Tue, 15 Sep 2009 15:16:26 GMT

Posted by: cesy

I just had an idea. Is it possible to collect statistics on how many users have chosen each of the system styles? Obviously it would need to exclude / account for custom styles in some way, and feed accounts which can't change their style. It would be interesting to see actual statistics on which styles and colour themes are most commonly used.

comments

New data sources wiki page

Sun, 13 Sep 2009 21:26:45 GMT

Posted by: foxfirefey

Edge data is live from last night's code push, folks! In celebration, I've made wiki pages on some Dreamwidth data sources:

http://wiki.dwscoalition.org/notes/Data_Sources

comments

Upcoming bug fix for ?auth=digest

Sat, 12 Sep 2009 20:39:41 GMT

Posted by: foxfirefey

There's currently a bug with an attachment for fixing ?auth=digest. (See this post for more info of the problem on LJ.) This should make it so that wget should work again--currently curl still does, though.

comments

Edges patch is committed!

Thu, 10 Sep 2009 03:00:13 GMT

Posted by: foxfirefey

After some further refinements by mark, edge patches have been committed! That means they're going out in the next code push! Exciting!

You can see the format here (I recommend JSONView) for viewing):

http://www.dev.memewidth.org/data/edges?user=amy

Names have been replaced with userids, which are then defined with username/type in a different struct.

comments

Interesting book on Natural Language Processing

Sat, 22 Aug 2009 07:17:53 GMT

Posted by: foxfirefey

For people who want to learn how to do natural language processing, the NLTK library has a CC'd book, Natural Language Processing with Python. Might be a useful resource for people wanting to make things based off of people's post content.

comments

Turn data/interests into JSON?

Thu, 20 Aug 2009 17:28:50 GMT

Posted by: foxfirefey

Now that fdata.bml is in JSON, would it be expedient to make interests.bml into JSON?

Pros: would be easier for Javascript mashups to load, is a standard format for parsing
Cons: would break compatibility with older LJ-based tools, would take up a bit more space than the current file.

comments

Let's talk about FOAF

Wed, 19 Aug 2009 01:26:55 GMT

Posted by: foxfirefey

All users have a FOAF file, like so:

http://foxfirefey.dreamwidth.org/data/foaf

There's a lot of useful data here, but I think it could be expanded to have some more data. ( Onward and out! )

comments

Upcoming fdata replacement: edges

Tue, 18 Aug 2009 02:41:02 GMT

Posted by: foxfirefey

So far Dreamwidth has been sorely lacking an equivalent to LJ's fdata, the data file that told you the relationships of a user with other users. Our new equivalent to that will be called edges. Because of the new complexity of DW's relationship structure, and due to a desire to use more standards, this data file will be in the JSON format. The bug for it is Bug 857.

I currently have my proposed patch in. It's not committed yet, or live on the site, but I think it probably will be within the next month. If you want a chance to prepare, I have my patch applied on my development server. It's possible that details will change with feedback, though, so be forewarned! My server has open registration, so you can feel free to create accounts and play around, but I've made a small set of test accounts such as this one. You can view an example edge data file here:

http://www.dev.memewidth.org/data/edges?account=amy

Here is an example of a community's edges file:

http://www.dev.memewidth.org/data/edges?account=bonny_bunnies

The data is in one big hash. There is an "account_type" variable, set to the type of the account. That makes it easy to determine what kind of account you are fetching data for. The other base variables are relationship types, such as "watched", "watched_by", or "member_of", that are a hash of arrays--all the accounts with a given relationship are split up into account types.

Any commentary on this planned data file? Problems you want to fix now, before it's implemented? Any plans for it you want to share?

comments

Which of your LJ friends are on DW?

Thu, 28 May 2009 18:39:29 GMT

Posted by: tobyaw

I’ve written a web page that looks at your LJ friends list, and tells you which of the usernames exist as accounts on Dreamwidth. Of course having the same username doesn’t necessarily mean they are the same person, but it might give you a head start in populating your Dreamwidth friends list.

http://lj-friends-on-dw.filmgold.com/

The implementation is naïve; there may be a more efficient way to see if a DW account exists. Source (and a command-line version) at http://github.com/filmgold/dw-tools

(Apologies if this isn’t an appropriate community to post this to — not absolutely clear from the community description.)

comments

This is how our site grows

Thu, 16 Apr 2009 01:43:50 GMT

Posted by: foxfirefey

Hello, memers! Boy, there sure are a lot more of you than there were two weeks ago, because of how much DW is growing. With the help of some scripts I set up a while ago and sophie's charting prowess, ( we've made a graph of exactly how much DW has grown lately. )

comments

LJMinion and Relationship Graphs

Sat, 11 Apr 2009 08:59:06 GMT

Posted by: foxfirefey

I've released the library that I made to create LJ relationship graphs. It uses NetworkX which uses matplotlib. It's fairly rudimentary, but sufficient for a simple graph of people an account has friended who've friended each other, with the size of the dots scaled to the number of mutual friends that person has.

( small example )

The functions in the package also provide some nice functionality for on the fly processing (ie, figuring out who do X and Y have friended in common). There's a fdata download and caching manager (default refresh rate: 2 weeks), and it rate limits its requests, so you don't have to be concerned about being hard on the servers when running it.

It can be found on BitBucket. I'm not too familiar with Python yet, though I have been enjoying it immensely, so anybody who Knows Better on how to do things is free to give me a guiding poke.

comments

A first step

Tue, 24 Mar 2009 05:51:41 GMT

Posted by: foxfirefey

As a side-shoot of my invite helper development, I made a quick stab at graphing LJ relationships. You can see examples here.

comments

A possible first use for Memewidth.org

Fri, 06 Mar 2009 22:48:41 GMT

Posted by: foxfirefey

So, at work I'm learning how to implement web services using Python and Django. I also want a small and manageable side project to help me learn things before I need them in the main project. I've thought of making an invite helper tool for this project. I'd host it on Memewidth. I know this has the possibility of being something that DW itself might want to integrate into the main project, and I wouldn't want that system to not have the data I collect, so I'll make sure that people know this information can be transferred to DW itself when they log in/sign up.

Does anybody have any thoughts about this? ~~Not sure what to name it, either.~~ Considering Saule for a name.

comments

Musings for a bot friendly profile info page

Thu, 05 Mar 2009 23:07:50 GMT

Posted by: foxfirefey

It might be nice to have a bot friendly profile page. It should contain the following information, assuming this information is judged to be okay to have easily bot accessible:

( List list list )

comments

Bot Policy

Sun, 22 Feb 2009 09:33:08 GMT

Posted by: foxfirefey

There is a proposed bot policy on the wiki from zorkian. It sounds pretty good; I especially like the bot domain--the ability to shut down extra server work during periods of high stress is a very nice one for both parties.

comments