Upcoming fdata replacement: edges

[foxfirefey]

So far Dreamwidth has been sorely lacking an equivalent to LJ's fdata, the data file that told you the relationships of a user with other users. Our new equivalent to that will be called edges. Because of the new complexity of DW's relationship structure, and due to a desire to use more standards, this data file will be in the JSON format. The bug for it is Bug 857.

I currently have my proposed patch in. It's not committed yet, or live on the site, but I think it probably will be within the next month. If you want a chance to prepare, I have my patch applied on my development server. It's possible that details will change with feedback, though, so be forewarned! My server has open registration, so you can feel free to create accounts and play around, but I've made a small set of test accounts such as this one. You can view an example edge data file here:

http://www.dev.memewidth.org/data/edges?account=amy

Here is an example of a community's edges file:

http://www.dev.memewidth.org/data/edges?account=bonny_bunnies

The data is in one big hash. There is an "account_type" variable, set to the type of the account. That makes it easy to determine what kind of account you are fetching data for. The other base variables are relationship types, such as "watched", "watched_by", or "member_of", that are a hash of arrays--all the accounts with a given relationship are split up into account types.

Any commentary on this planned data file? Problems you want to fix now, before it's implemented? Any plans for it you want to share?

Comments

[zvi]

My (extremely cursory) understanding of JSON is that one of the reasons it is attractive is that it's human readable.

The long hash isn't really human readable, I think it would be good to put some whitespace in.

[foxfirefey]

That's a bit out of the scope of this bug--the most important thing is that scripts will be able to easily get the information; prettification options can be added later. This is a lot more compact and saves space.

[birguslatro]

Does this help? (hopefully I've got it right)...

example edge data file...

{
    "maintainer": {
        "personal": [ "beth" ]
    },
    "moderator": {},
    "member": {
        "personal": [ "amy", "beth" ] 
    },
    "watched_by": {
        "personal": [ "amy", "beth", "caitlin" ]
    },
    "account_type": "community"
}

example of a community's edges file...

{
    "trusted_by": {
        "personal": [ "ellen", "gina" ]
    },
    "trusted": {
        "personal": [
            "beth", "caitlin", "ellen", "fiona", "gina", "hayley",
            "ingrid", "janice", "kelly", "margaret", "nina"
        ]
    },
    "watched": {
        "community": [
            "anophalous_ages", "bonny_bunnies", "every_elephant",
            "fickle_feelings"
        ],
        "personal": [
            "beth", "caitlin", "danni", "ellen", "fiona", "gina", "hayley",
            "ingrid", "janice", "kelly", "leala", "nina"
        ]
    },
    "watched_by": {
        "personal": [ "ellen", "fiona", "gina" ]
    },
    "member_of": {
        "community": [
            "anophalous_ages", "bonny_bunnies", "cautious_coneys",
            "fickle_feelings", "great_grapes"
        ]
    },
    "account_type": "personal"
}

[foxfirefey]

Okay, so apparently I wasn't using the right MIME type for JSON anyway--it's application/json. This makes Firefox prompt you for a download. However, there is a FF extension to make it all pretty instead:

https://addons.mozilla.org/en-US/firefox/addon/10869

So I think that should be sufficient.

[cesy]

What kind of things use this data?

[foxfirefey]

I want to do this, except for DW. Other things I could think of would be an icons of your circle collage, figuring out what interests you share in common with your circle members, what interests they have in common that you don't have, seeing what journals or communities or feeds you had the most circle people watching that you don't.

[cesy]

Oh, that would be so cool.

[sofiaviolet]

seeing what journals or communities or feeds you had the most circle people watching that you don't

So, like popwithfriends.bml? No lie, I was pondering a suggestion that DW implement some kind of Popular with Your Subscriptions feature.

[foxfirefey]

I don't think that would be a bad idea, really, and pretty easy to implement. I think I'd know how to implement it myself. Wanna write it up for suggestions?

[sofiaviolet]

Sure. :)

[birguslatro]

What does this give you that the FOAF file doesn't? Just a more parser-friendly file?

[foxfirefey]

Tons more. Our FOAF has all the people's journals you subscribe to in foaf:Know tags, but that's it. Communities aren't there, nor OpenID users or feeds watched. And it doesn't say who you've given access to, who you have access from, or who is subscribed to you.

Additionally, FOAF is comparatively huge to a smaller format like this, which I think makes a difference when you have to data mine umpteen gazillion of them.

[birguslatro]

Ah - that makes sense. And FOAF I assume has to be more generic, whereas this can be specifically tailored to DW.

Carry on! :)

[foxfirefey]

FOAF doesn't have to be more generic, but it's a lot more overhead and work to add to. We'd have to add a namespace for DW stuff.

One nice thing about the JSON format is that Javascript programs can natively load the file as an object, I think.

[birguslatro]

I nearly commented before about similarities in format between JSON and REBOL, (my language of choice), but I didn't. But I've just watched a video of a talk JSON's designer gave on the history of JSON, and he mentioned REBOL as an influence, so I wasn't just seeing things! It's a good talk too, so have a look if you've not already seen it...

http://developer.yahoo.com/yui/theater/video.php?v=crockford-json

A full transcript's there too.

(REBOL's problem is it isn't open-source - makes you weep, it does.)