• Announcements

    • MannDude

      Current state of vpsBoard   02/04/2017

      Dear vpsBoard members and guests:

      Over the last year or two vpsBoard activity and traffic has dwindled. I have had a change of career and interests, and as such am no longer an active member of the web hosting industry.

      Due to time constraints and new interests I no longer wish to continue to maintain vpsBoard. The web site will remain only as an archive to preserve and showcase some of the great material, guides, and industry news that has been generated by members, some of which I remain in contact to this very day and now regard as personal friends.

      I want to thank all of our members who helped make vpsBoard the fastest growing industry forum. In it's prime it was an active and ripe source of activity, news, guides and just general off-topic banter and fun.

      I wish all members and guests the very best, whether it be with your business or your personal projects.

      -MannDude
wlanboy

wlanboy.com - how it started

16 posts in this topic

So this is the short story about wlanboy.com - a twitter archive dedicated to vps providers.

 

Twitter and all other social services are banned on work so I needed a remote twitter client.

Main reason was about missing campaigns.

Yup - I missed some good coupon codes and therefore wanted to build a tool that is searching for them.

And - of course - one that updates me on any deals.

 

Next feature was about igonring dinky tweets (one without likes, retweets,etc) - and to find new twitter accounts that might be interesting (followers/following/rt).

After some time I added lists to manage the stored profiles and tweets.

A list is a group of twitter accounts that are sharing the same topic.

Like:

  • .Net
  • Java
  • VPS providers
  • Redis / MongoDB
  • Ruby / Gems
  • etc

A profile stands for an twitter user. It does include information about how often he/she writes something to ensure that the workers are not polling profiles that do not have any new information. And about the quality of the tweets. So a profile can be deactivated if someone is only tweeting about his dog.

 

The system itself is build out of following components:

  • A Ruby thin app that is providing the webpage
  • A MongoDB cluster holding all data
  • A RabbitMQ cluster which is load balancing and distributing workloads
  • A cronjob that is creatinig workloads
  • A bunch of workers that are listening on RabbitMQ queues for work

A simple cronjob (Ruby script) is looping through all profiles to check if a twitter account should be updated.

It is creating a workload item (including the latest tweet id) and sends it to a RabbitMQ topic.

One of the workers is fetching the work order, is scraping the twitter profile, is storing all new tweets and is updating the profile.

 

Currently only the "VPS provider" list is public and can be accessed through wlanboy.com.

 

Counting through all lists (today) about 3700 twitter profiles are stored in the database.

The "VPS provider" list does include 37 active profiles (and about 20 disabled ones).

 

After the homepage was finished I added some additional services:

  • full text search with and without profile name
  • RSS feed for all new added feeds
  • Statistics about the number of tweets per day

Yesterday I switched the whole domain to SSL only.

It was about time todo that after the switch of vpsboard.

Was quite a hassle to switch everything to local files to ensure that every file is based on my own domain (even the fonts referenced in css).

 

Today I will start to publish my tutorials on my homepage too (for the ones who want them without the IP.Board html bloat).

For me it is exciting to see what can be made out of a simple terminal script - if you don't stop after the second month.

9 people like this

Share this post


Link to post
Share on other sites

Nice insight. Hopefully others will write about their motives and the structure of their projects too. I think this post is a invitation to ask questions. I want to start with some :)

 

Question 1: How do you parse the websites? What libs do you use?

 

Question 2: Why do you not make money with all the information and tutorials?

 

Question 3: Why do you choosed Mongodb? Hype?

Share this post


Link to post
Share on other sites
Question 1: How do you parse the websites? What libs do you use?

 

Question 2: Why do you not make money with all the information and tutorials?

 

Question 3: Why do you choosed Mongodb? Hype?

  1. HTML is XML so I am using xpath to pick the elements I want.

    Need a tool to get xpaths out of a current website?

    Use the Firefox Plugin Firebug.

    On the HTML view you can select any element and right click on it to copy the xpath for this element.

     

  2. Most of my knowledge is based on free resources.

    Public libraries and free online stuff. People like me writing tutorials about things they manage to run.

    I don't invent groundbreaking stuff - I only editing and enhancing allready known knowledge.

    Maybe making it (hopefully) more accessible to beginners.

    It would be simply not ok to make money with that.

     

  3. No hype - it just worked for my interface. I decided to use json for all communication.

    It is easier to add stuff.

    It does have some drawbacks but the advantages outbalance them.

3 people like this

Share this post


Link to post
Share on other sites

You deserve at least a hug for #2

No purple so thanks for the compliment.

Share this post


Link to post
Share on other sites

Thank you for the hint to use xpath.

People who copy others work or draw some free vps to get people writing for them should read #2 two times. I hate the people with their ad polluted howto websites with outdated and never testet content.

1 person likes this

Share this post


Link to post
Share on other sites

People who copy others work or draw some free vps to get people writing for them should read #2 two times. I hate the people with their ad polluted howto websites with outdated and never testet content.

Was that a subtle remark towards me?

Share this post


Link to post
Share on other sites

Was that a subtle remark towards me?

It was my anger about a lot of howto websites.Not every point applies to you. You do not steal content.

But you are selling ads

fVitKx6i5ko2kXa.jpg

Share this post


Link to post
Share on other sites

It was my anger about a lot of howto websites.Not every point applies to you. You do not steal content.

But you are selling ads

fVitKx6i5ko2kXa.jpg

And so are vpsBoard, I don't see the difference?

Share this post


Link to post
Share on other sites

But you are selling ads

Please calm down.

Mikho is collecting tutorials and naming each author - so I do not see anything bad here.

His website is good and his Wordpress Theme is well done. I know myself that it is not easy to do a site like this with Wordpress.

I would like to see more people collecting knowledge - hey even I started to collect my own stuff.

And it is totally fine to place ads if it helps him to cover costs.

1 person likes this

Share this post


Link to post
Share on other sites

Mikho please accept my apology for my harsh words.

3 people like this

Share this post


Link to post
Share on other sites

Mikho please accept my apology for my harsh words.

apology accepted.

2 people like this

Share this post


Link to post
Share on other sites

Updated the whole layout to start a more simplistic approach with normalize.css and font-awesome.css. 

Share this post


Link to post
Share on other sites

  • Similar Content

    • By wlanboy
      For me planning a homepage does include three steps:
      plattform framework management I would like to discuss what current options are available for these topics. Basically if I am still on the right path.
      Back to the topic:
      Management Do I need a GUI for creating content? Dynamic or static content? Framework Site creators or CMS Plattform vps server webhosting About 10 years ago I started with the plattform (budget) and afterwards thought about the frameworks.
      Nowadays my first question is if it has to be dynamic, second one if it can be avoided to be dynamic.
      Long story short: I am done with about any CMS. Content management systems brake on about anything from upgrades, themes and plugins. Don't even think you can customize anything without slapping yourself a few months later.
      Yooomla, Badpress, Ghosty, Drupi or Redexe are broken by design. I stopped using them and I stopped hosting them.
      My tutorials, my blog, my documentation are all written in md. Markdown helps a lot if you want to generate something out of it. No style, only content.
      I started with Jekyll, moved to Octopress (due to the addons - it is more like a Jekyll++), tried Pelican and Middleman, moved to Hyde and now moved everything to Hugo.
      There is still a little bit of Ruby flying aroung, because searching and paging (if you do not want to generate 1000nd of html pages) is still the last resort of dynamic content generation. But even wlanboy.com is 99% static content with two dynamic views.
      After about 15 years of php, phyton and ruby frameworks I feel quite well with:
      markdown to preserve content static site generators to get css/js injected html ruby & phyton for some searching/paging My next plan is to get rid of ruby & phyton on the frontend side.
      100% generated webpage with some js code which is doing the searching, paging, tagging stuff through a rest-service:

      - [js] - [js rest-client] - [rest-service] - [database]. Simple html only hosting for all webpages and frontends and some javascript clue that is handling the dynamic stuff. The rest-service is hiding the database. So no SQL-injection and an easy move if I want to switch databases.
      There will be something like a 100% dynamic management frontend to create content, but that does not have to be installed on the webservers. Good luck to hack a static html page on a get-only webserver without any write or execution rights.
       
      I like to have my homepages on my dropbox. Generating new content on my laptop and it is automatically published - a small ruby script ftp-ing the html/css/js stuff.
      write markdown file with your texteditor of choice start static site generator of your choice syncing generated folder (html files) to your server/hosting account done Perfect separation of content (markdown), theme (css/jss/layout) and output (generated html).
       
    • By wlanboy
      https://vimeo.com/87249046
      So what did I build?
       
      My servers <-> Internet <-> Rapsberry Pi <-> Arduino <-> Shiftregister <-> blinking things
       
      Cronjob running a Ruby script
      This script does: Load list of ips and loop through list doing: Ping ip Try to login via ssh (key based) Check if other sessions are open Check if logfiles should be checked (logwatch report with bad words) Save short status to byte array Write this array to the EEPROM (512 bytes!!) of the Arduino (serial) Loop within Arduino Load array of status bits Loop through array For each item Increase counter and refresh digit display (count + 1) Check status of each byte Enable or disable LEDs So the digit display is showing which server status is currently displayed (0 to 9 - so 10 servers).
      The 6 LEDs (red/green/red/green/yellow/yellow) do show:
      Ping status (red bad / green good) Service status (red bad / green good) [if every required service is running or not] Other sessions are open (yellow) [so someone is playing around] Logstatus (yellow) [Hey I should look at those log files]  
      I am still searching for a good 16 bit shift register (maybe the STP16C596A) and I want to add a 4x16 LCD module too.
       
      The Raspberry Pi is able to handle the stuff too but can only work with 3.3V input/output where the Arduino can handle 5V input/output.
      It is a pain to work with resistors for each single GPIO of the Raspberry Pi:

       
      And - of course - you save a lot of energy because the Arduino is more energy efficient.
       
      I like the idea of separated areas of responsibility:
      Raspberry Pi doing all the network and analysing stuff and Arduino doing all the electro-technics.
       
      I toy with the idea of buying a USB hub to connect several Arduinos with one Raspberry Pi.
      One Arduino for the LCD/LED stuff
      One Arduino for the temperature/light/movement stuff
      One Arduino for a touch button pad - each button for one action like ssh login and calling scripts, etc.
       
      Yup your right - I am currently running riot on this electro stuff.
       
       
      Edit: Added video link.
    • By wlanboy
      This thread is all about posting private homepages. Maybe some other board members might run a homepage on their vps too. I am looking forward for links.
       
      So I will start with wlanboy.com - tweeter archive.
      .
      All done in Ruby running on a Thin cluster living on a 96MB vps from ServerDragon.
      The MongoDB is running on a Ramnode SSD vps.
       
      Main purpose: Twitter archive of all vps related twitter accounts.
      Twitter and all other social services are banned on work so I needed a remote twitter client. After some time a search/RSS feed was added too.
       
      PS: If a twitter account is missing - just PM me. I don't want to miss one.