Social Contract

As we start the new year, a big question on my mind is: what will become of our relationships with people we don't know, people who aren't like us?

It's tempting to feel like the social contracts in place to ensure we aren't blatantly reckless with each other's lives, health or humanity have started to untangle just recently. But I suppose that would be my privilege talking.

If I get a little too bummed that it feels like people are no longer bothering to balance their self-interests with the greater good, I try to remember that Black people, people of color, women, poor people and many other populations have been affected by of some version of that imbalance for generations, and have often suffered greatly because of it.

Still, there's nothing like being headed toward a preventable year three of a pandemic to crystalize the ways  in which so many people seem to be saying an active "screw you" to any notion of addressing a public health crisis through even minimal personal sacrifice. And I too often feel a great big "screw you back" welling up inside of me in response, as much as I wish it weren't so.

I'm working on that, but I'm not always sure if working on it means embracing that energy and letting it inspire me to action, or suppressing it in favor of more diplomatic responses. (You know, for the greater good.)

I guess these are all versions of the questions I asked in June 2020:

  • How can we get people to care about and act on problems that don’t seem to immediately or directly affect them?
  • How can we get people to care about making the distinction between what’s verifiably true and what’s not?
  • How do we make change happen fast enough to matter?
  • What should I do?

There is a great deal of good in the world, and a great deal of good in my own life. I have so much to be thankful for.

But I used to believe that People are, on the whole, good and wanting good things for each other. Now I'm not so sure, and I don't like that uncertainty. It feels like it could poison the future. My future.

I'd like to raise my daughter with an optimistic outlook about the nature of humanity, but I may need to scale that back a bit. An optimistic outlook about the good she can create in her own life and for those around her? An optimistic outlook about next week's dinner menu?

Happy New Year.

Unlocking community event information from Facebook

It wasn't always this way.

It might be hard to remember, but there was a time when Facebook wasn't the only place to learn about upcoming events in our communities. Instead of having to scroll past silly memes and political rants to get the details of the next potluck, lecture or book club you cared about, the information was available in lots of other places. Events were available on websites and apps, and they were shared via constantly updated feeds that you could even integrate directly into your own personal calendaring system.

You could learn about what was happening down the street or across town, and you didn't have to give up your online privacy to do it.

At some point, this shifted. Weary of the duplication of efforts maintaining event information in multiple places, and increasing fragmentation of sources that you needed to consult to figure out what's happening, people longed for a more centralized, authoritative spot to enter and learn about community events. I was one of those people.

Enter Facebook

Facebook saw this opportunity and jumped on it.

They made it easy to enter and promote events, and they sprinkled in social features to make it extra compelling to do. Not only could you learn about the event itself, you could see who among your friends was planning on (or interested in) going, and what they had to say about it. If you were an event organizer you could see what kinds of events were catching the interest of your target audience, and you could more easily avoid scheduling your big fundraising gala on the same night as the art museum open house.

At first, Facebook made access to event data one of the most friction-free parts of its platform, and they didn't require you to be "on Facebook" to see it. You could get email notifications, RSS feeds, ICS feeds to bring event information into your own productivity tools and daily life. There was API access so that you could build tools and websites that incorporated the event data where you wanted it to be.

Over time, all of that access was removed. The APIs were shut off, the feeds were shut down, and the message was clear: you have to come to the Facebook website or app to learn about events that you might want to attend.

It's nothing new to note that Facebook's user interface and business model are built around keeping people inside its walled garden. Whatever you might want to contribute to or get from your Facebook experience, they want you to do it on their website or in their mobile app, and on their terms. Their ability to sell advertising space depends on it.

But as with many other aspects of Facebook culture and its grip on our personal and community data, there's a significant downside. Facebook's decision to lock up event information has real implications for how we encounter and experience public life in our real-world communities.

Why is this a problem?

It means that like our exposure to news or updates from our online friends, our awareness of community events is driven by a black-box algorithm optimized around profit over everything else. It means that one company's shifting views on what constitutes an acceptable event, and its sensitivity to the interests of paying advertisers and political organizations could determine whether we see the details of an upcoming protest, demonstration or other exercising of free speech.

And it means that anyone concerned about the privacy implications of having their interest in a given event tracked, sold and monetized may have less exposure to the events that may have traditionally shaped and defined public life in their community, and private life in their circles of friends and neighbors.

There are practical implications too. An organization or business that still chooses to have its own website is having to enter and maintain event information in multiple places, which is time consuming and inefficient. People who want to avoid using Facebook are either left out of the loop or forced into using it again.

I went looking for solutions to this in the context of a website project I was working on recently, where the request was simple: can we bring our Facebook event data into our WordPress site? I had naively assumed that Facebook would have an interest in making this easy: if people could enter their events in one place and have them pushed out to everywhere that mattered, it would be so much easier to see them as the natural place to maintain that information.

But again, they don't make it easy. There's no API available to fetch event data, even for a Facebook page on which I'm an administrator. Event data is not displayed on public-facing Facebook.com pages in any kind of structured ways, and in fact it is rendered in ways that make it resistant to traditional "scraping" tools. There are no other user- or developer-friendly tools for working with event data that I can find.

Yes, Facebook does have a "page widget" that lets you display event information from a Facebook page elsewhere via embedding, if you are an admin on that page. The layout and customization options are pretty limited, but more importantly this is not the same thing as having access to the event information itself for importing, displaying, searching, archiving or other actions that someone might want to take.

Asking page owners to initiate displaying the event information elsewhere also eliminates the chance for other interested parties to "slice and dice" event data in potentially useful or interesting ways. If I want to create a website where you can search and display all of the animal adoption-related events in the Facebook facebook happening within 50 miles of my zip code, I can't. I can't do this even if all of the animal adoption agencies enter their events on Facebook, make them public and put page widgets up on their own website.

Can software fix this?

Being a software developer who works all day long on tools that try to make publishing easier and more interconnected, as I thought about this issue I still felt like there must be some way to extract event data from publicly shared Facebook events. After all, if you can see them in your web browser even while not logged in to Facebook, then that means the data is by definition publicly accessible in some form.

After analyzing the structure of a Facebook web page, its Javascript and the asynchronous HTTP calls made to fully render the content on it, I found that there is a way. Facebook's own event display pages are making a POST to the URL https://www.facebook.com/api/graphql/  to retrieve relevant event details, which are then rendered in HTML and CSS for a normal user to see. I sniffed the request and removed all of the query parameters I could find that might be extraneous. Here's an example of a resulting logged-out POST request that retrieves a batch of the upcoming events for a given Facebook page (my local Parks and Recreation Department):

POST /api/graphql/ HTTP/1.1
Host: www.facebook.com
Content-Type: application/x-www-form-urlencoded
Origin: https://www.facebook.com
User-Agent: PostmanRuntime/7.15.2
Accept: */*
Cache-Control: no-cache
Host: www.facebook.com
Accept-Encoding: gzip, deflate
Content-Length: 255
Connection: keep-alive
cache-control: no-cache

fb_api_req_friendly_name=PageEventsTabUpcomingEventsCardRendererQuery&variables=%7B%22pageID%22%3A%2250650939390%22%7D&__req=8&__user=0&av=0&__a=1&__be=1&dpr=2&fb_api_caller_class=RelayModern&__pc=PHASED%3ADEFAULT&__comet_req=false&doc_id=2343886202319301

I can't tell you what all of the query parameters mean but I think doc_id is the Facebook-internal indicator of a shorthand for which query to run (so apparently "2343886202319301" maps to "all upcoming events" or something like that) and pageID is the internal ID of the Facebook page itself. I can imagine this query breaking later on if Facebook decided to update the doc_id mappings, but for the time being, it works.

The result of the request is a JSON object that contains all the upcoming event info one would need to import Facebook events into another system, display them on a non-Facebook website, and so on.

From this point, I could imagine creating a tool that, given a list of Facebook pages, automatically and regularly grabs all of the upcoming public events available in Facebook and does something useful with them. Here's a proof of concept PHP script that lays the groundwork for that.


<?php
/**
* Proof of concept, retrieve publicly-available Facebook page event data.
* Chris Hardie <chris@chrishardie.com>
*
* To use, first add Guzzle as a dependency:
* $ commposer require guzzlehttp/guzzle
*
*/
require 'vendor/autoload.php';
use GuzzleHttp\Client;
define( 'FB_UPCOMING_EVENT_DOC_ID', 2343886202319301 );
$facebook_pages = array(
'50650939390' => array(
'page_name' => 'Richmond Parks Department',
'page_url' => 'https://www.facebook.com/richmondparks/',
'category' => 'outdoor',
),
);
$facebook_base_params = array(
'fb_api_req_friendly_name' => 'PageEventsTabUpcomingEventsCardRendererQuery',
'__req' => '8',
'__user' => '0',
'av' => '0',
'__a' => '1',
'__be' => '1',
'dpr' => '2',
'fb_api_caller_class' => 'RelayModern',
'__pc:PHASED' => 'DEFAULT',
'__comet_req' => 'false',
);
$client = new Client(
[
// Base URI is used with relative requests
'base_uri' => 'https://www.facebook.com',
// You can set any number of default request options.
'timeout' => 2.0,
]
);
$headers = [
'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36',
'Content-Type' => 'application/x-www-form-urlencoded',
'Origin' => 'https://www.facebook.com',
];
foreach ( $facebook_pages as $facebook_page_id => $facebook_page ) {
$facebook_variable_value = '{"pageID":"' . $facebook_page_id . '"}';
$facebook_form_params = array_merge(
$facebook_base_params,
array(
'variables' => $facebook_variable_value,
'doc_id' => FB_UPCOMING_EVENT_DOC_ID,
)
);
$upcoming_event_response = $client->request(
'POST',
'/api/graphql/',
[
'headers' => $headers,
'form_params' => $facebook_form_params,
]
);
$response_code = $upcoming_event_response->getStatusCode();
if ( ! empty( $response_code ) && ( 200 === $response_code ) ) {
$body = $upcoming_event_response->getBody();
if ( null !== $body ) {
$fb_response = json_decode( $body, true );
$fb_events = $fb_response['data']['page']['upcoming_events']['edges'];
}
}
if ( ! empty( $fb_events ) && ( 0 < count( $fb_events ) ) ) {
foreach ( $fb_events as $fb_event ) {
$event = $fb_event['node'];
echo $event['id'] . PHP_EOL;
echo $event['name'] . PHP_EOL;
echo $event['time_range']['start'] . PHP_EOL;
echo $event['event_place']['contextual_name'] . PHP_EOL;
echo $event['event_place']['city']['contextual_name'] . PHP_EOL;
echo $event['event_buy_ticket_url'] . PHP_EOL;
echo PHP_EOL;
}
}
}
exit;

Am I going to get in trouble for this?

Does Facebook care if we access event data in this way? Yes, yes they do. Their terms of service about Automated Data Collection explicitly states that if you try to extract data from the Facebook platform in an automated way, they can ban you forever. But more likely than anything is that they will change the way their platform works to make this kind of data retrieval even more difficult.

I think there's hope for some shifting expectations about what's considered fair here. A U.S. appeals court ruled just last month that web scraping does not constitute illegal activity, and went so far as to acknowledge that if a scraper is retrieving publicly available data owned not by the platform but by its users, the scraping can't be blocked.

In the case of Facebook event data, it's worth noting again that we're talking about publicly available information shared by organizations that want it to get more exposure online about events that they want the public to attend. Restricting access to that because it's a potential source of advertising revenue seems beyond greedy to me. In the end though, it's up to organizations, businesses and individuals to decide whether they want to have their event data locked up, or out on the open web.

You can help

If you are an event organizer, please consider posting your event data outside of Facebook on a publicly available website!

If you are someone who cares about building a healthy culture of civic engagement in your community, advocate for the organizations you're involved with to move away from tools that make this harder in the long term!

If you are a developer at Facebook or anywhere else that builds tools for people to share information, please don't lock up that information! Show your commitment to the open web. Provide APIs, RSS (or in this case, ICS) feeds, good documentation and facilitate easy exports of user-owned data. (A quick shout-out to the folks at Eventbrite who, at least for now, make available for free a very robust API to access community events shared on their platform.)

I'm glad for any feedback and suggestions about these challenges; please comment below. I've also explored the themes discussed here in various past posts, including:

 

Article on Strong Towns

I've been enjoy and appreciating the articles on Strong Towns for a while now. They bring a provocative and practical long-term view to the conversation about how cities and towns in North America can become strong and resilient. Their mission and approach line up so well with the advocacy work I've done in my own community, and the thinking that I see leading to great results in communities around the world.

Now it's an honor to have one of my own essays on community building appear in Strong Towns. I wrote about how to get a response from local leaders when you need to get something done. Most of what I write on RichmondMatters.com is specific to life in Richmond, Indiana, but in this case it turned out that with a few refinements to the original article, the information might be relevant to a national audience.

I hope it helps someone out there find a way forward on positive change they're pursuing in their own town.

Remote workers want community too

I've been spending more time with people who do most their work remotely. Since writing about the advantages of distributed/remote teams versus working in person, I've been paying attention to all of the time remote workers spend figuring out how to be around other people in just the right doses. They're looking to be in the same place with some fellow humans who have a common sense of purpose, or who at least share an understanding of the remote work lifestyle, even if just for a little while.

This leads me to wonder if the future of remote work isn't just a bunch of people on their own, working from home offices or coffeeshops, but instead an arrangement of remote workers coming together in person for a sense of shared experience. Ironically, it might even end up looking a lot like traditional notions of where and how people work, but probably with a lot more freedom, flexibility and fun along the way.

Here are some of the signs I see:

Continue reading "Remote workers want community too"

Shutting down Richmond Brainstorm

20140322-screenshotFive years ago this month I launched the community improvement website RichmondBrainstorm.com. The site allowed users to submit ideas for ways to make Richmond, Indiana a better place, allowed other users to discuss and vote on those ideas, and shared success stories of ideas that had been implemented. I created the site because I think it's important for a given community to shape its own course for the future instead of waiting for solutions from state and national governments, and because I was tired of hearing good, creative ideas from others that never seemed to get the attention or visibility they deserved.

In the time since, some 86 community improvement ideas were submitted and discussed, and a number of the ideas became real projects that were implemented. The site got over 140,000 visits from around 45,000 unique visitors. I've also received contact from people other communities around the country asking for help to create a similar resource in their city, and so the idea of an online community improvement idea inventory seems to itself have become an idea worth spreading.

But, after an initial period of significant activity, the Richmond Brainstorm site had become largely dormant, with no new ideas submitted to it in close to a year. Over the years I've regularly talked to local community development organizations who have said the concept of the site is an exciting one and could even be integrated into their own efforts at prompting further conversations and action, but as yet Richmond does not seem to be a place where most of those kinds of conversations want to happen online, for better or worse. That combined with the time that it takes to keep the site's software current, deal with spammers and perform other administrative tasks has begun to outweigh the value that I think RichmondBrainstorm.com is currently bringing to the community.

So, as of today I'm shutting the site down.

Continue reading "Shutting down Richmond Brainstorm"

Disruptors among us

Security Theater KnifeIt seems that every healthy and thriving community, city, social group or ecosystem remains healthy and thriving because they can handle having a disruptive element with an important role to play.

The disruptors are the people or events that shake up the status quo, question why things are done a certain way, or introduce an element of chaos or discontent into the system and forces it to evolve, change or adapt.  They are the people whose survival or well-being is not dependent on the stability of the system; revolutionary new ideas and significant change rarely come from those whose livelihoods and sense of security depend on things going along as they are.

Continue reading "Disruptors among us"

A city in distress

edvard munch - the scream  1893If you live in or near Richmond, Indiana, it looks like you picked the wrong year to stop sniffing glue. (Does anyone get Airplane! references anymore?)

Have you ever scored yourself on the Holmes and Rahe stress scale, which measures your personal stress based on which of 43 major life events (death, divorce, job change, etc) have happened to you in the past year? It's an interesting scale because it recognizes that significant life change - positive or negative - brings with it an increased potential for illness and possibly other problems.

I think if we were to do a similar scoring of events in the life of a midwestern city, Richmond, Indiana would be somewhere in the "freaking out" to "going ape-shit" range. I dare say, we're a city in distress.

Even beyond what's happening at the state, national and global level - economic turmoil, war and other violence, toxic political races, Charlie Sheen's career, etc. - I suggest that the last year in the life of our community has been an unusually tumultuous one here.

Just a partial list of some major events I've observed in the recent life of Richmond in no particular order:

Continue reading "A city in distress"

An anonymous letter about the state of Richmond

Last month I received an anonymous and wide-ranging letter in the mail about the state of affairs in Richmond, Indiana, addressed to "Positive Place Committee, Madame Mayor, Richmond and Wayne County Government Officials, Palladium Item Advisory Board, and Leadership pundits."

I take it that I received it because I'm on the Palladium-Item's community editorial advisory board (though I would much rather reside in the 'leadership pundit' category because it sounds cooler).  The letter was mailed on May 15th, and was sent via USPS to an incorrect version of my office address, but made it to me anyway.

Continue reading "An anonymous letter about the state of Richmond"

7 Ways to Help Young Professionals Engage in Richmond

IMG_2224.JPGThere's a new group in town - H.Y.P.E. Richmond - that is working to "connect and mobilize young professionals to make the Richmond area an even greater place to live, work, and play."  If you're interested in those efforts, you might consider joining in on the brainstorming session they're having tonight at the Firehouse BBQ restaurant, 5:30 to 7 PM.

I won't be able to attend, but as an employer of some younger professionals who gets to hear some of their concerns and struggles "engaging" in life in Richmond, and as someone who has spent my own young professional life in Richmond, I want to offer a few initial ideas about how to help connect and mobilize that demographic.  (This is in addition to the ideas already being submitted and discussed at RichmondBrainstorm.com.)  My hope is that others will add to the list over time:

Continue reading "7 Ways to Help Young Professionals Engage in Richmond"