This commit is contained in:
Stevan Freeborn committed
1 parent
2f19ca0d09
commit
5540a492ca
1 file changed
+16
@@ -12,6 +12,8 @@ I've spent a good amount of my adult life making things on the internet. It star
|
|||||||
|
|
||||||
However, the internet of today isn't the same as it was back then and I imagine that will continue to be the case as time goes on. Platforms come, many go, and some stay and even those that stay change and evolve over time losing and increasing relevance depending on whatever else is going on in the world. A pain point for me as someone who has a regular life with a day job, kids, and a wife has always been that I've found it quite a bit exhausting to always have to be maintaining the different identities I have across platforms like [Twitter](https://x.com/stevanfreeborn), [Facebook](https://www.facebook.com/stevanreece), [Instagram](https://www.instagram.com/stevan.freeborn/), [YouTube](https://youtube.stevanfreeborn.com), [Twitch](https://twitch.stevanfreeborn.com), etc. Not to mention it is a pain that anything I share on one of those only exists there and anyone who follows me on one of those only follows me there. The social web and let's be honest for the most part today the social web is pretty much a bunch of siloed, walled gardens that sit next to each other. From a content creation perspective as well as just from a general user perspective this sucks. If I ever want to stop using one platform it means I stop getting the content from the people I enjoy following and the small subset who enjoy following me don't get any of my content either. Granted I know this isn't quite the way content gets distributed to folks on these platforms today, but I think you get my point. Our identities and our data on these apps exist only and exclusively to those apps.
|
However, the internet of today isn't the same as it was back then and I imagine that will continue to be the case as time goes on. Platforms come, many go, and some stay and even those that stay change and evolve over time losing and increasing relevance depending on whatever else is going on in the world. A pain point for me as someone who has a regular life with a day job, kids, and a wife has always been that I've found it quite a bit exhausting to always have to be maintaining the different identities I have across platforms like [Twitter](https://x.com/stevanfreeborn), [Facebook](https://www.facebook.com/stevanreece), [Instagram](https://www.instagram.com/stevan.freeborn/), [YouTube](https://youtube.stevanfreeborn.com), [Twitch](https://twitch.stevanfreeborn.com), etc. Not to mention it is a pain that anything I share on one of those only exists there and anyone who follows me on one of those only follows me there. The social web and let's be honest for the most part today the social web is pretty much a bunch of siloed, walled gardens that sit next to each other. From a content creation perspective as well as just from a general user perspective this sucks. If I ever want to stop using one platform it means I stop getting the content from the people I enjoy following and the small subset who enjoy following me don't get any of my content either. Granted I know this isn't quite the way content gets distributed to folks on these platforms today, but I think you get my point. Our identities and our data on these apps exist only and exclusively to those apps.
|
||||||
|
|
||||||
|
## What is a Personal Data Server (PDS)?
|
||||||
|
|
||||||
It was because of this reason I got really excited about the [At protocol](https://atproto.com/) when I learned about it. I signed up for [Bluesky](https://bsky.app/) likely for a lot of the same reasons everyone else did. Twitter became some strange, lesser form of its former self and if you liked that style of social media Bluesky seemed to present a great alternative experience. At the time though I knew nothing about the underlying technology or how absolutely kickass the whole vision of it was. See the idea behind the At protocol is basically that the social web should be like the original web was - open, decentralized, interoperable. Users should be able to establish an identity on the network and then own and control the content they publish on that network. I'm a fairly new beginner when it comes to all things atproto and many smarter people have done a better job writing about what the At protocol is, how it works, and how you can try to conceptually think about it so I'll just link to those folks below.
|
It was because of this reason I got really excited about the [At protocol](https://atproto.com/) when I learned about it. I signed up for [Bluesky](https://bsky.app/) likely for a lot of the same reasons everyone else did. Twitter became some strange, lesser form of its former self and if you liked that style of social media Bluesky seemed to present a great alternative experience. At the time though I knew nothing about the underlying technology or how absolutely kickass the whole vision of it was. See the idea behind the At protocol is basically that the social web should be like the original web was - open, decentralized, interoperable. Users should be able to establish an identity on the network and then own and control the content they publish on that network. I'm a fairly new beginner when it comes to all things atproto and many smarter people have done a better job writing about what the At protocol is, how it works, and how you can try to conceptually think about it so I'll just link to those folks below.
|
||||||
|
|
||||||
- [https://atproto.com/guides/understanding-atproto](https://atproto.com/guides/understanding-atproto)
|
- [https://atproto.com/guides/understanding-atproto](https://atproto.com/guides/understanding-atproto)
|
||||||
@@ -22,6 +24,8 @@ The point of this post though is to talk through a recent project I did related
|
|||||||
|
|
||||||
The [Personal Data Server](https://atproto.com/specs/account) is a part of the [At protocol specification](https://atproto.com/specs/atp) which means anyone can build their own implementation of it and then anyone is free to host those implementations and users are free to pick and choose where they specifically choose to host their data on one of these PDSs. You can even host your own on your own hardware as I've chosen to do and how I did it is what I want to share with you. This likely won't serve as a great step by step tutorial given the topology of my home lab probably is unique to me, but maybe some of what I share can help you and at the very least pique your interest into learning more about the At protocol and getting involved with it.
|
The [Personal Data Server](https://atproto.com/specs/account) is a part of the [At protocol specification](https://atproto.com/specs/atp) which means anyone can build their own implementation of it and then anyone is free to host those implementations and users are free to pick and choose where they specifically choose to host their data on one of these PDSs. You can even host your own on your own hardware as I've chosen to do and how I did it is what I want to share with you. This likely won't serve as a great step by step tutorial given the topology of my home lab probably is unique to me, but maybe some of what I share can help you and at the very least pique your interest into learning more about the At protocol and getting involved with it.
|
||||||
|
|
||||||
|
## Choosing a PDS Implementation
|
||||||
|
|
||||||
The first step in setting up a PDS is deciding on which implementation you are going to use. There are actually quite a few floating around in the wild, but the vast majority from what I looked at seem to be more of toy or hobby projects rather than anything serious that you'd actually want to deploy and run in production. There are though some options and I'll just share the ones I considered below:
|
The first step in setting up a PDS is deciding on which implementation you are going to use. There are actually quite a few floating around in the wild, but the vast majority from what I looked at seem to be more of toy or hobby projects rather than anything serious that you'd actually want to deploy and run in production. There are though some options and I'll just share the ones I considered below:
|
||||||
|
|
||||||
- [Bluesky PDS](https://github.com/bluesky-social/pds)
|
- [Bluesky PDS](https://github.com/bluesky-social/pds)
|
||||||
@@ -30,6 +34,8 @@ The first step in setting up a PDS is deciding on which implementation you are g
|
|||||||
|
|
||||||
Ultimately I chose to stick with what looks to be the oldest, most used, and probably best supported and maintained - which to no surprise - is Bluesky's own implementation. I know it is a pretty boring choice, but when it comes to picking technologies that I am actually going to trust to support my own personal needs for the foreseeable future boring and reliable sound wonderfully attractive to me. Not to mention it never hurts to be using something a lot of other people are also using and which likely means there is a lot of freely available helpful information about out on the web...because you know the web is awesome like that. Kind of illustrates the point why you'd want the social web to be similar.
|
Ultimately I chose to stick with what looks to be the oldest, most used, and probably best supported and maintained - which to no surprise - is Bluesky's own implementation. I know it is a pretty boring choice, but when it comes to picking technologies that I am actually going to trust to support my own personal needs for the foreseeable future boring and reliable sound wonderfully attractive to me. Not to mention it never hurts to be using something a lot of other people are also using and which likely means there is a lot of freely available helpful information about out on the web...because you know the web is awesome like that. Kind of illustrates the point why you'd want the social web to be similar.
|
||||||
|
|
||||||
|
## Deployment Architecture & Reverse Proxy Setup
|
||||||
|
|
||||||
With a PDS implementation selected it is now time to figure out the plan for deployment. This is where things are going to get uber specific to me so let me just give you the lay of the land a bit for myself when it comes to self-hosting things like this. I like to use VPSs either hosted in my homelab or in someone's cloud. I like to proxy traffic from the outside world to my servers through [Cloudflare](https://cloudflare.com). I like to use [Nginx](https://nginx.org/) as a webserver. And whenever possible I like to run things in containers on the host. I don't think any of that is edgy or really notable, but those technologies are ubiquitous, well-supported, and don't have much of a cost to them at all.
|
With a PDS implementation selected it is now time to figure out the plan for deployment. This is where things are going to get uber specific to me so let me just give you the lay of the land a bit for myself when it comes to self-hosting things like this. I like to use VPSs either hosted in my homelab or in someone's cloud. I like to proxy traffic from the outside world to my servers through [Cloudflare](https://cloudflare.com). I like to use [Nginx](https://nginx.org/) as a webserver. And whenever possible I like to run things in containers on the host. I don't think any of that is edgy or really notable, but those technologies are ubiquitous, well-supported, and don't have much of a cost to them at all.
|
||||||
|
|
||||||
I don't want to get into the details too much on the DNS and networking side of things here, but I do want to highlight what steps I took in preparation just to make sure I had things properly configured to make sure that traffic could reach my server. I started by making sure that there was a DNS record entry for the domain I want the PDS to be available at pointing to the machine where the PDS container will be running. Then I made sure I had an nginx configuration setup to properly route traffic received at the machine for this domain to the PDS which will be running in a container. Below is an example of what that config looks like:
|
I don't want to get into the details too much on the DNS and networking side of things here, but I do want to highlight what steps I took in preparation just to make sure I had things properly configured to make sure that traffic could reach my server. I started by making sure that there was a DNS record entry for the domain I want the PDS to be available at pointing to the machine where the PDS container will be running. Then I made sure I had an nginx configuration setup to properly route traffic received at the machine for this domain to the PDS which will be running in a container. Below is an example of what that config looks like:
|
||||||
@@ -75,6 +81,8 @@ server {
|
|||||||
|
|
||||||
Pretty standard typical Nginx reverse proxy stuff. Seriously I don't mean to skip over anything here just everyone's exact deployment environment is likely going to be different especially when we get into the self-hosting world that I don't want to belabor any of it more than what is necessary. If you have questions you can always hit me up on the atmosphere here.
|
Pretty standard typical Nginx reverse proxy stuff. Seriously I don't mean to skip over anything here just everyone's exact deployment environment is likely going to be different especially when we get into the self-hosting world that I don't want to belabor any of it more than what is necessary. If you have questions you can always hit me up on the atmosphere here.
|
||||||
|
|
||||||
|
## Running PDS in Docker
|
||||||
|
|
||||||
The [Bluesky PDS](https://github.com/bluesky-social/pds) implementation already comes containerized and if you are starting with a fresh machine they've pretty much done all the heavy lifting for you with great install and setup [script](https://github.com/bluesky-social/pds#installing-on-ubuntu-200422042404-and-debian-111213) you can use to get up and running in no time aside from any potential networking or DNS management you might need to do on your own. In my case though I'm not deploying to a fresh machine. The machine I'm using is a shared host that I use for several other things and is already setup to handle traffic proxied through Cloudflare and has Nginx already running on it as a reverse proxy so using the out-of-the-box setup that Bluesky provides didn't work for me. The changes though I had to make were really minimal.
|
The [Bluesky PDS](https://github.com/bluesky-social/pds) implementation already comes containerized and if you are starting with a fresh machine they've pretty much done all the heavy lifting for you with great install and setup [script](https://github.com/bluesky-social/pds#installing-on-ubuntu-200422042404-and-debian-111213) you can use to get up and running in no time aside from any potential networking or DNS management you might need to do on your own. In my case though I'm not deploying to a fresh machine. The machine I'm using is a shared host that I use for several other things and is already setup to handle traffic proxied through Cloudflare and has Nginx already running on it as a reverse proxy so using the out-of-the-box setup that Bluesky provides didn't work for me. The changes though I had to make were really minimal.
|
||||||
|
|
||||||
I basically stole the necessary services configurations for their official docker compose file and created one for myself that looks like this:
|
I basically stole the necessary services configurations for their official docker compose file and created one for myself that looks like this:
|
||||||
@@ -168,6 +176,8 @@ curl https://<proper_domain>/xrpc/_health
|
|||||||
wsdump "wss://<proper_domain>/xrpc/com.atproto.sync.subscribeRepos?cursor=0"
|
wsdump "wss://<proper_domain>/xrpc/com.atproto.sync.subscribeRepos?cursor=0"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Creating Your Account & Domain Handle
|
||||||
|
|
||||||
If all those check out you can be pretty confident you now have your very own PDS that you can use for all your things atproto related. However first you'll need to establish an account on the PDS, perform the needful to claim the handle you want, and actually start using that account and identity to do things on the atmosphere.
|
If all those check out you can be pretty confident you now have your very own PDS that you can use for all your things atproto related. However first you'll need to establish an account on the PDS, perform the needful to claim the handle you want, and actually start using that account and identity to do things on the atmosphere.
|
||||||
|
|
||||||
Lucky for us the pds container contains all the tools you need for the job, specifically this CLI tool called [`goat`](https://github.com/bluesky-social/goat). This is a tool that enables you to do all the necessary bits of administration you'll need to do for creating/updating accounts, creating invite codes, etc. In this case we can keep it pretty simple and get our first account created like so:
|
Lucky for us the pds container contains all the tools you need for the job, specifically this CLI tool called [`goat`](https://github.com/bluesky-social/goat). This is a tool that enables you to do all the necessary bits of administration you'll need to do for creating/updating accounts, creating invite codes, etc. In this case we can keep it pretty simple and get our first account created like so:
|
||||||
@@ -196,8 +206,14 @@ docker exec pds goat did resolve-handle --handle <username>.open.social
|
|||||||
|
|
||||||
If that was successful you can be confident you've got an account that is hosted on your PDS and whose data will be stored on your PDS, but whose identity can be used all around the atmosphere. Easiest way to test is just by signing in to your account using [Bluesky](https://bsky.app). You'll need to make sure that you specifically set the hosting provider to your pds using the domain you configured to reach it at.
|
If that was successful you can be confident you've got an account that is hosted on your PDS and whose data will be stored on your PDS, but whose identity can be used all around the atmosphere. Easiest way to test is just by signing in to your account using [Bluesky](https://bsky.app). You'll need to make sure that you specifically set the hosting provider to your pds using the domain you configured to reach it at.
|
||||||
|
|
||||||
|
## Operations: Email Relay & Backups
|
||||||
|
|
||||||
Couple of things I want to touch on related to this set up. The first is related to allowing my PDS to send emails. The PDS implementation supports this by supplying the necessary information as the following environment variables: `PDS_EMAIL_SMTP_URL`, `PDS_EMAIL_FROM_ADDRESS`, and `PDS_CONTACT_EMAIL_ADDRESS`. The underlying host I deployed my PDS to is already configured to send emails using a relay service so I just set up a local account for the PDS to use for this purpose, but you can obviously choose the solution that best fits your setup and needs. The second is that I opted to mount a TrueNAS dataset to the host machine and use a dead simple shell script and cron job to regularly back up the locations that the PDS is storing data in as specified by these environment variables: `PDS_DATA_DIRECTORY` and `PDS_BLOBSTORE_DISK_LOCATION`. I think this is an important consideration given that you are now 100% in control and thus responsible for the safety and retention of the data you publish to the atmosphere as it all ends up back here on your PDS. I can just hear Uncle Ben now...happy to share more specifics about these two things if you have questions. Reach out to me [here](https://bsky.app/profile/stevanfreeborn.com).
|
Couple of things I want to touch on related to this set up. The first is related to allowing my PDS to send emails. The PDS implementation supports this by supplying the necessary information as the following environment variables: `PDS_EMAIL_SMTP_URL`, `PDS_EMAIL_FROM_ADDRESS`, and `PDS_CONTACT_EMAIL_ADDRESS`. The underlying host I deployed my PDS to is already configured to send emails using a relay service so I just set up a local account for the PDS to use for this purpose, but you can obviously choose the solution that best fits your setup and needs. The second is that I opted to mount a TrueNAS dataset to the host machine and use a dead simple shell script and cron job to regularly back up the locations that the PDS is storing data in as specified by these environment variables: `PDS_DATA_DIRECTORY` and `PDS_BLOBSTORE_DISK_LOCATION`. I think this is an important consideration given that you are now 100% in control and thus responsible for the safety and retention of the data you publish to the atmosphere as it all ends up back here on your PDS. I can just hear Uncle Ben now...happy to share more specifics about these two things if you have questions. Reach out to me [here](https://bsky.app/profile/stevanfreeborn.com).
|
||||||
|
|
||||||
|
## Migrating an Existing Account
|
||||||
|
|
||||||
Last thing I want to touch on before wrapping up is on account migration because so far we've only really gone through the necessary steps for establishing a new PDS with a new account, but chances are if you are already on Bluesky you'll want to also migrate that existing account to this new PDS. Originally I thought this was going to be a topic for a whole separate blog post which it definitely is if you want to go the manual route. Lucky for us though the manual route isn't the only option anymore. There is this really wonderful tool called [PDS MOOver](https://pdsmoover.com/) created by a gentleman named [Bailey Townsend](https://bsky.app/profile/pds.dad) that will do all the work for you and if I had to guess with far more consideration for everything that can go wrong than you'll be able to muster yourself. The tool also has support for maintaining a backup of your PDS data for you as well which can help alleviate some of the pressure you may have felt from my comments above regarding being responsible for keeping your data safe. I'm using it myself. Again if you run into any troubles yourself using the tool feel free to reach out to me [here](https://bsky.app/profile/stevanfreeborn.com) and I'll be happy to try and steer you in the right direction.
|
Last thing I want to touch on before wrapping up is on account migration because so far we've only really gone through the necessary steps for establishing a new PDS with a new account, but chances are if you are already on Bluesky you'll want to also migrate that existing account to this new PDS. Originally I thought this was going to be a topic for a whole separate blog post which it definitely is if you want to go the manual route. Lucky for us though the manual route isn't the only option anymore. There is this really wonderful tool called [PDS MOOver](https://pdsmoover.com/) created by a gentleman named [Bailey Townsend](https://bsky.app/profile/pds.dad) that will do all the work for you and if I had to guess with far more consideration for everything that can go wrong than you'll be able to muster yourself. The tool also has support for maintaining a backup of your PDS data for you as well which can help alleviate some of the pressure you may have felt from my comments above regarding being responsible for keeping your data safe. I'm using it myself. Again if you run into any troubles yourself using the tool feel free to reach out to me [here](https://bsky.app/profile/stevanfreeborn.com) and I'll be happy to try and steer you in the right direction.
|
||||||
|
|
||||||
|
## Conclusion
|
||||||
|
|
||||||
Alright I've taken up more than enough of your time today with my atproto excitement and retelling, but I hope I've sparked your interest even a little about what atproto has to offer both users and content creators and maybe even enough to get you to start getting involved with it yourself. I know things can look pretty grim right now for the general state of the internet, but to me atproto presents a really exciting future for what could be and I'd like to try to do my part to help make it so by using it, talking about it, and sharing it with others.
|
Alright I've taken up more than enough of your time today with my atproto excitement and retelling, but I hope I've sparked your interest even a little about what atproto has to offer both users and content creators and maybe even enough to get you to start getting involved with it yourself. I know things can look pretty grim right now for the general state of the internet, but to me atproto presents a really exciting future for what could be and I'd like to try to do my part to help make it so by using it, talking about it, and sharing it with others.
|
||||||
Reference in new issue
Block a user