wlanboy
Content Contributer
So this is the short story about wlanboy.com - a twitter archive dedicated to vps providers.
Twitter and all other social services are banned on work so I needed a remote twitter client.
Main reason was about missing campaigns.
Yup - I missed some good coupon codes and therefore wanted to build a tool that is searching for them.
And - of course - one that updates me on any deals.
Next feature was about igonring dinky tweets (one without likes, retweets,etc) - and to find new twitter accounts that might be interesting (followers/following/rt).
After some time I added lists to manage the stored profiles and tweets.
A list is a group of twitter accounts that are sharing the same topic.
Like:
A profile stands for an twitter user. It does include information about how often he/she writes something to ensure that the workers are not polling profiles that do not have any new information. And about the quality of the tweets. So a profile can be deactivated if someone is only tweeting about his dog.
The system itself is build out of following components:
A simple cronjob (Ruby script) is looping through all profiles to check if a twitter account should be updated.
It is creating a workload item (including the latest tweet id) and sends it to a RabbitMQ topic.
One of the workers is fetching the work order, is scraping the twitter profile, is storing all new tweets and is updating the profile.
Currently only the "VPS provider" list is public and can be accessed through wlanboy.com.
Counting through all lists (today) about 3700 twitter profiles are stored in the database.
The "VPS provider" list does include 37 active profiles (and about 20 disabled ones).
After the homepage was finished I added some additional services:
Yesterday I switched the whole domain to SSL only.
It was about time todo that after the switch of vpsboard.
Was quite a hassle to switch everything to local files to ensure that every file is based on my own domain (even the fonts referenced in css).
Today I will start to publish my tutorials on my homepage too (for the ones who want them without the IP.Board html bloat).
For me it is exciting to see what can be made out of a simple terminal script - if you don't stop after the second month.
Twitter and all other social services are banned on work so I needed a remote twitter client.
Main reason was about missing campaigns.
Yup - I missed some good coupon codes and therefore wanted to build a tool that is searching for them.
And - of course - one that updates me on any deals.
Next feature was about igonring dinky tweets (one without likes, retweets,etc) - and to find new twitter accounts that might be interesting (followers/following/rt).
After some time I added lists to manage the stored profiles and tweets.
A list is a group of twitter accounts that are sharing the same topic.
Like:
- .Net
- Java
- VPS providers
- Redis / MongoDB
- Ruby / Gems
- etc
A profile stands for an twitter user. It does include information about how often he/she writes something to ensure that the workers are not polling profiles that do not have any new information. And about the quality of the tweets. So a profile can be deactivated if someone is only tweeting about his dog.
The system itself is build out of following components:
- A Ruby thin app that is providing the webpage
- A MongoDB cluster holding all data
- A RabbitMQ cluster which is load balancing and distributing workloads
- A cronjob that is creatinig workloads
- A bunch of workers that are listening on RabbitMQ queues for work
A simple cronjob (Ruby script) is looping through all profiles to check if a twitter account should be updated.
It is creating a workload item (including the latest tweet id) and sends it to a RabbitMQ topic.
One of the workers is fetching the work order, is scraping the twitter profile, is storing all new tweets and is updating the profile.
Currently only the "VPS provider" list is public and can be accessed through wlanboy.com.
Counting through all lists (today) about 3700 twitter profiles are stored in the database.
The "VPS provider" list does include 37 active profiles (and about 20 disabled ones).
After the homepage was finished I added some additional services:
- full text search with and without profile name
- RSS feed for all new added feeds
- Statistics about the number of tweets per day
Yesterday I switched the whole domain to SSL only.
It was about time todo that after the switch of vpsboard.
Was quite a hassle to switch everything to local files to ensure that every file is based on my own domain (even the fonts referenced in css).
Today I will start to publish my tutorials on my homepage too (for the ones who want them without the IP.Board html bloat).
For me it is exciting to see what can be made out of a simple terminal script - if you don't stop after the second month.
Last edited by a moderator:
