amuck-landowner

Coding contest: Community driven ping/traceroute project

wlanboy

wlanboy

Content Contributer
We all know the status/looking glass scripts you can add to your vps. It is nice to offer this to visitors but the benefit is quite small because it is a single destination and even if someone wants to start a ping/traceroute from your location he/she has to find it first.

So the idea is to create a distributed ping/traceroute script:
* Centralized database containing the list of nodes
* A admin frontpage to add new nodes to this database

* A page displaying the list of all verified nodes
* A client frontpage to select nodes and to start pings/traceroutes
* Afterwards add services like dns lookup, etc

For security reasons only the frontpage has the rights to add new nodes.
But all other nodes do have the right to get the list of other nodes.

So everyone offering a node is able to run a site offering the ping/traceroute/etc services of all other nodes.
And this will be the main benefit of this project and the benefit to offer a node.

Now the contest starts.
I talked to ServerDragon and they are offering 2 free OpenVZ servers paid for a whole year as a "thank you" for the persons doing the programming work.
The result of this contest will be open source. So everyone can start his own ping "community". During the test phase I will host the frontpage and the database. I like the idea to have a vpsBoard service, so afterwards it will be hosted by vpsBoard. And everyone can add a node if he/she wants to.

The project itself does have four steps:
1. Presenting the idea
2. Discussion on how it should be done
3. Implementation
4. Testing & going life

Well I do not know if this idea is great. And I am not sure how it should be implemented. I guess doing it with PHP will lower the barrier to someone wanting to add a node.
The frontend is quite simple. Just fill in all required information to add a note. And a single page returning the list of all approved nodes (maybe with search parameters).
The node itself will present a list of available nodes. The user can select ... up to 20 nodes ... and is starting the requests and afterwards presenting the results.
All communication is done via HTTP. So only the frontend needs a database connection. The data itself is transported via JSON.

The database will contain following information of a node:

version number of script, ip, network, hoster, city, country.

Looks like I have reached the end of step1. Now step 2 starts.
What do you think about this project? Did I missed something? Should something be done in another way?

After step 2 is done we will split the work into parts. Everyone can offer his hands to implement one of the parts. The best submission for each part will win one of the free vps.
 

Of course every submitter will get credits for this project. I have started a blank GitHub project: https://github.com/wlanboy/vpsboard.com-dping

Feel free to add/change whatever you want.
 
NodeBytes

NodeBytes

Dedi Addict
I've already begun work on something like this... it would be distributed across many nodes as well. 

http://vpsboard.com/index.php?/topic/406-multi-location-traceroute-project/

As far as what I am working on in the coding, it is in PHP and Python so far. The idea here, as well as yours, is to be able to spread it across as many locations as possible. I see this as being beneficial to the community as well. I do however think that the database should not be centralized to one server, I think possibly cloning the database may be better as far as redundancy goes, we don't want the whole service to fail if one server goes down. I will put more thought into that tomorrow.
 
wlanboy

wlanboy

Content Contributer
I like the idea behind the torrent trackers. One central communication point to collect nodes.

We can use GitHub as backup. The node database will dump the node table to GitHub once a day.

Because the admin site is returning the node list (http://ip-of-admin/nodelist) as json  the database dump will be json too.

So one single point to add nodes. And a backup of the database holding the list of all nodes.

Everyone can decide if he/she wants to use the master or the github file.
 
wlanboy

wlanboy

Content Contributer
And everyone would be able to fork his own nodelist too.

Noone will have single access to any information. But we need one single entry point to build up something like a community.

Tracker: http://nodeping.vpsboard.com

The database containing one single table and the "register your node" page will be hosted well ... on the vps of the contest winner.

If vpsBoard fanishes ... well someone else will start a new register your node page.

If the contest winner fanishes ... well vpsboard will link the subdomain to a new node.

The new hoster of the database/register page will just take the last backup and restarts the project.

And if you do not like vpsBoard you have to live with the database backup on your own.
 
Last edited by a moderator:
W

walesmd

New Member
This actually sounds like a fun project to take on... are we just tossing around ideas now? Do we have any established dates for certain milestones to be hit? Is there a list of criteria that will be judged and/or minimally required feature set? I think the end of June to at least have entries submitted for judging is a fair amount of time.

Either way, count me in.
 
wlanboy

wlanboy

Content Contributer
We are now discussing how it should be done. After collecting all input we are defining the milestones. Last step would be counting the people doing the work.

Critical workflow part is the client side call of the commands:

  1. visitor opens start page of node
  2. JQuery.GET to get json list of possible nodes
  3. start page displays input field for ip/host and well ... a list of possible hosts
    How should a huge list of nodes be displayed?
    List of countries in dropdown?
    Second list of cities in auto filled dropdown?

    Additional:
    List of countried in dropdown?
    Second list of providers in auto filled dropdown?

    Some JS to fill the dropboxes on selection of others...

    Or should be do call the master again to display only the nodes matching the search parameters?
  4. User selects nodes out of a list of nodes he wants (checkboxes?)
  5. User clicks go button
  6. JQuery.GET to all selected nodes
  7. Fill <pre>s in table with all responses (JQuery)
I do not want to have any db connection or sockets or ssh calls on client side. All nodes do have to support a http get call that will only return blank result information (no html).
URI for node list: http://master/nodelist?country=xx&city=yy&provider=ww

URI for node action: http://node/action/ping?host=127.0.0.1

Client side would be just JQuery and plain html. No Python/Ruby proxy server.

We have to check for a good regex too to ensure that the parameters do only contain ip/hosts. I do not want to have something like "host=127.0.01&&rm -Rf/" :)
 
W

walesmd

New Member
Please forgive me if I misunderstood - is this a contest or a community project? I don't see how a single person can rightly be identified for the prize if it's a community effort. Beyond that, there's a whole lot of stipulations at a very technical level there; it sounds like contract work in which the payment is VPS usage. There's absolutely nothing wrong with that, it's just not a contest.
No one asked, but I'll share, how I would run something like this:

  • Stick with post #1 but lose some of the details. Details stifle innovation - we can't think outside of the box when you're pouring concrete over it on all sides.

    We are looking for an application to run as a vpsBoard branded service. We all know and love the Looking Glass scripts, unfortunately they only test from one location and it's a pain in the ass trying to track them all down. You're mission: develop the next generation in Looking Glass script, worthy of the vpsBoard brand, that will carry the community into the foreseeable future. You are free to implement features as you choose but all entries must at least meet the following functional requirements:

    All functionality displayed on-screen should function as described. Incomplete features made available to the end user will be judged negatively.
  • Endpoint node owners must have the ability to easily add/remove themselves from the "Master Node List" at will.
  • The "Master Node List" must be implemented in such a way as to eliminate a single point of failure.
  • [Add other, generic, required features as you envision the end result]
  • [The goal is to establish a framework for the end product while encouraging innovation]
[*]Furthermore, all entries must meet these administrative requirements:

  • Entries must be licensed under a GPL-compatible license.
  • Entries may only use GPL-compatible libraries/packages to which they are licensed and have appropriately identified.
  • Entries must include full installation instructions.
  • If selected as the winning entry, all contestants must be willing and able to configure and install their entry on the vspBoard Service VPS.
  • Contestants must use GitHub, clone the repo, develop publicly, yeah yeah other stuff.
[*]Setup a baseline GitHup repository that all entries must publicly fork - which you've already done. This allows people to use the Network tab and see who is competing / what progress is being made. Additionally, all contestants must include a file named VPSBOARD.md within the root of their repository, this file should link back to their vpsBoard.com profile (2-way verification that it's really me when paired with a thread on this forum, linking to my repo).
[*]Establish a date, roughly half-way through, for "technical document" submission. These documents do not have to be finalized or impressive in anyway as they are not judged; they should convey the overall technical stack the entry uses and some of the key features the developer would like to highlight. Within 3 days of this date, each judge must privately respond to these documents. This is a final opportunity for a developer to reconsider their design decisions and make sure they are headed in the right direction. This does not open a communication line to the developers and there shall be no further communication, in regards to this contest, between the two groups.
[*]Establish an end date for the contest as a whole. On, or before, this date all contestants must tag their release as 'vpsBoard-submission'. This is the repository judges will clone and make their judgements against. Developers are not required to provide a "full stack" solution (for instance, load balancing, cacheing, etc). The web application itself, along with the application's data layer, are the judgeable submission requirements.
[*](optional? We probably all have available hosting, but this would require that we do - barrier to entry) Additionally, within 3 days of contest end, each contestant must host a version of their application should the judges be unable (or unwilling, given the developer's design decisions) to get the application up and running on their own. Judge's use of this hosted version should not, in itself, factor into the judging criteria; but ease of installation, appropriate use of technologies, etc should be taken into account.
[*]Within the same 3-day span as provided for application hosting, developers should ensure all documentation is in order and their entry is available as open source software appropriately licensed. As a courtesy to future users and the general public, developers must clearly state their intent to further develop and support their entry - this decision is not judged within the competition.
 
Last edited by a moderator:
wlanboy

wlanboy

Content Contributer
BlueVM said:
I've already done this:

https://github.com/mojeda/ServerStatus
Click to expand...
Wow - really cool stuff.

If we would choose your project as base following things are missing:

  • Form to pull for an action:
    - ping


    - traceroute
  • Selectable list of node that should perform the action
  • Decentralized "list of nodes that are available"
All scripts that are available today do provide information for a defined list of nodes that are owned by the person who is maintaining the script.

Our goal is to have a decentraclized list of nodes that can be used to perfom pings/traceroutes.

@walesmd

Your right. It might be a community project. But this community is quite small. In my opinion there would not be a lot of people that a) would like the idea and b) would spend their time on it. So I searched for a small "thank you" for 1-3 people doing the stuff. Additionally a contest is something that attracts attention maybe even beyond the borders of this community.

Currently there is only one guy saying "i will help" - well you. And if there will be such a small group of people doing the stuff we will not need any bureaucracy.

But maybe I am wrong with my point of view on how many php developers we will catch. Your plan sounds reasonable and I will follow it. Hopefully we catch a lot devs to start a community project.

But back to the list of open items:

  1. Communication protocol node <-> node
  2. A way to implement a failsave node list
  3. A way to manage this list of nodes

The list of nodes is public available. So my nodes will be part of a list. Once downloaded, forked, or whatever. Only way to remove it is to change my URI of the service. E.g. mydomain.com/123pinger to mydomain.com/nodepinger123. The maintainer of the nodelist will check the nodes once a day. A 404 will delete the entry. A timeout will mark the node as offline. 3 marks and the node is deleted.

Any other open items?
 
acd

acd

New Member
I'm interested to see how you folks plan to mitigate abuse/peer authenticate/limit query rates. Writing an unauthenticated (open) peer ping/trace daemon in C is like a weekend project (10-15 hrs), probably less in a scripting language using existing tools. Anything but the most basic access controls significantly increases the complexity.
 
wlanboy

wlanboy

Content Contributer
acd said:
I'm interested to see how you folks plan to mitigate abuse/peer authenticate/limit query rates.
Click to expand...
Yes you are right. Additionally we have to pass an argument to the console so we have to double/triple check that this string does not contain more than an ip or domain name.
 
BlueVM

BlueVM

New Member
Verified Provider
@wlanboy - Its licensed under WTFPL. My next version would probably be better for the task as it has a full template engine, remote status scripts, icmp, history, graphs an administration panel, user and administrator notifications, etc...
 
HalfEatenPie

HalfEatenPie

The Irrational One
Retired Staff
I saw version 2.0 of Telephone's looking glass. 

Looks fantastic and I believe it does everything we've been trying to make here? 
 
BlueVM

BlueVM

New Member
Verified Provider
@mojeda - I'd volenteer to help, but I've got a large number of projects as it is and some of the stuff listed here doesn't fit into my plans for the server status system...
 
mojeda

mojeda

New Member
BlueVM said:
@mojeda - I'd volenteer to help, but I've got a large number of projects as it is and some of the stuff listed here doesn't fit into my plans for the server status system...
Click to expand...
Thanks for the offer, I understand that you have other things to work on. Not sure if you are also helping with Neon, but that's something I'm looking forward to!
 
You must log in or register to reply here.
Top
amuck-landowner