wlanboy.com - how it started

Discussion in 'The Pub (Off topic discussion)' started by wlanboy, Jan 15, 2014.

  1. wlanboy

    wlanboy Content Contributer

    2,126
    1,169
    May 16, 2013
    So this is the short story about wlanboy.com - a twitter archive dedicated to vps providers.


    Twitter and all other social services are banned on work so I needed a remote twitter client.


    Main reason was about missing campaigns.


    Yup - I missed some good coupon codes and therefore wanted to build a tool that is searching for them.


    And - of course - one that updates me on any deals.


    Next feature was about igonring dinky tweets (one without likes, retweets,etc) - and to find new twitter accounts that might be interesting (followers/following/rt).


    After some time I added lists to manage the stored profiles and tweets.


    A list is a group of twitter accounts that are sharing the same topic.


    Like:

    • .Net
    • Java
    • VPS providers
    • Redis / MongoDB
    • Ruby / Gems
    • etc

    A profile stands for an twitter user. It does include information about how often he/she writes something to ensure that the workers are not polling profiles that do not have any new information. And about the quality of the tweets. So a profile can be deactivated if someone is only tweeting about his dog.


    The system itself is build out of following components:

    • A Ruby thin app that is providing the webpage
    • A MongoDB cluster holding all data
    • A RabbitMQ cluster which is load balancing and distributing workloads
    • A cronjob that is creatinig workloads
    • A bunch of workers that are listening on RabbitMQ queues for work

    A simple cronjob (Ruby script) is looping through all profiles to check if a twitter account should be updated.


    It is creating a workload item (including the latest tweet id) and sends it to a RabbitMQ topic.


    One of the workers is fetching the work order, is scraping the twitter profile, is storing all new tweets and is updating the profile.


    Currently only the "VPS provider" list is public and can be accessed through wlanboy.com.


    Counting through all lists (today) about 3700 twitter profiles are stored in the database.


    The "VPS provider" list does include 37 active profiles (and about 20 disabled ones).


    After the homepage was finished I added some additional services:

    • full text search with and without profile name
    • RSS feed for all new added feeds
    • Statistics about the number of tweets per day

    Yesterday I switched the whole domain to SSL only.


    It was about time todo that after the switch of vpsboard.


    Was quite a hassle to switch everything to local files to ensure that every file is based on my own domain (even the fonts referenced in css).


    Today I will start to publish my tutorials on my homepage too (for the ones who want them without the IP.Board html bloat).


    For me it is exciting to see what can be made out of a simple terminal script - if you don't stop after the second month.
     
    Last edited by a moderator: Sep 2, 2016
    Abdussamad, Epidrive, juan and 6 others like this.
  2. peterw

    peterw New Member

    800
    188
    Jun 14, 2013
    Nice insight. Hopefully others will write about their motives and the structure of their projects too. I think this post is a invitation to ask questions. I want to start with some :)

    Question 1: How do you parse the websites? What libs do you use?

    Question 2: Why do you not make money with all the information and tutorials?

    Question 3: Why do you choosed Mongodb? Hype?
     
  3. wlanboy

    wlanboy Content Contributer

    2,126
    1,169
    May 16, 2013
    1. HTML is XML so I am using xpath to pick the elements I want.
      Need a tool to get xpaths out of a current website?


      Use the Firefox Plugin Firebug.


      On the HTML view you can select any element and right click on it to copy the xpath for this element.
    2. Most of my knowledge is based on free resources.
      Public libraries and free online stuff. People like me writing tutorials about things they manage to run.


      I don't invent groundbreaking stuff - I only editing and enhancing allready known knowledge.


      Maybe making it (hopefully) more accessible to beginners.


      It would be simply not ok to make money with that.
    3. No hype - it just worked for my interface. I decided to use json for all communication.
      It is easier to add stuff.


      It does have some drawbacks but the advantages outbalance them.
     
    vRozenSch00n, peterw and Shados like this.
  4. tchen

    tchen New Member

    276
    124
    Sep 1, 2013
    You deserve at least a hug for #2
     
  5. wlanboy

    wlanboy Content Contributer

    2,126
    1,169
    May 16, 2013
    No purple so thanks for the compliment.
     
  6. peterw

    peterw New Member

    800
    188
    Jun 14, 2013
    Thank you for the hint to use xpath.

    People who copy others work or draw some free vps to get people writing for them should read #2 two times. I hate the people with their ad polluted howto websites with outdated and never testet content.
     
    vRozenSch00n likes this.
  7. mikho

    mikho Not to be taken seriously, ever!

    963
    253
    May 15, 2013
    Was that a subtle remark towards me?
     
  8. peterw

    peterw New Member

    800
    188
    Jun 14, 2013
    It was my anger about a lot of howto websites.Not every point applies to you. You do not steal content.

    But you are selling ads

    [​IMG]
     
    Last edited by a moderator: Apr 30, 2017
  9. mikho

    mikho Not to be taken seriously, ever!

    963
    253
    May 15, 2013
    And so are vpsBoard, I don't see the difference?
     
    Last edited by a moderator: Apr 30, 2017
  10. wlanboy

    wlanboy Content Contributer

    2,126
    1,169
    May 16, 2013
    Please calm down.

    Mikho is collecting tutorials and naming each author - so I do not see anything bad here.

    His website is good and his Wordpress Theme is well done. I know myself that it is not easy to do a site like this with Wordpress.

    I would like to see more people collecting knowledge - hey even I started to collect my own stuff.

    And it is totally fine to place ads if it helps him to cover costs.
     
    vRozenSch00n likes this.
  11. peterw

    peterw New Member

    800
    188
    Jun 14, 2013
    Mikho please accept my apology for my harsh words.
     
    vRozenSch00n, mikho and Shados like this.
  12. mikho

    mikho Not to be taken seriously, ever!

    963
    253
    May 15, 2013
    apology accepted.
     
    vRozenSch00n and peterw like this.
  13. vRozenSch00n

    vRozenSch00n New Member

    256
    79
    May 17, 2013
    I love vpsboard and the good people  :wub:
     
    mikho likes this.
  14. wlanboy

    wlanboy Content Contributer

    2,126
    1,169
    May 16, 2013
    Back to topic:

    The first bunch of tutorials are archived.
     
  15. wlanboy

    wlanboy Content Contributer

    2,126
    1,169
    May 16, 2013
    Updated the whole layout to start a more simplistic approach with normalize.css and font-awesome.css. 
     
  16. buildmyblock

    buildmyblock New Member

    12
    0
    Sep 13, 2016
    nice work i like the website its extremely useful keep up the good work :)