Register to post in forums, or Log in to your existing account
 

Play RetroMUD
Post new topic  Reply to topic     Home » Forums » CMUD General Discussion
TesterOfLimitz
Novice


Joined: 02 Jun 2007
Posts: 37

PostPosted: Mon Mar 02, 2009 6:56 am   

Gathering information from a webpage?
 
I was hoping to write a script that checks Lusternia's 'Players online' every five to ten minutes, just to see what the general playerbase is like at any given hour of the day. I can use 'qw' in game to check this, but the problem lies with the fact I'd have to be signed in, and being AFK is against the rules.... HOWEVER, luckily for me, you can see this on their homepage:

Players Online: 209

So I was curious to see if there was any way for cMud to internally open the website, and gather the number (which seems unlikely) or if there was any way to open the webpage in a new firefox tab, gain the number of online players, then close said tab right after (which seems a little less unlikely, buuut hopefully possible!)

Any and all help would be appreciated, and I look forward to seeing if this is actually viable!
Reply with quote
gamma_ray
Magician


Joined: 17 Apr 2005
Posts: 496

PostPosted: Mon Mar 02, 2009 11:42 am   
 
This is called "page scraping". Google has lots of information, but I've never done anything like this personally. One fairly rudimentary way that comes to mind to get this done is to invoke wget (or similar) from CMud every 5-10 minutes, and then use CMud to parse the resulting text file.
Reply with quote
Fang Xianfu
GURU


Joined: 26 Jan 2004
Posts: 5155
Location: United Kingdom

PostPosted: Mon Mar 02, 2009 11:51 am   
 
No, but yes. There's no way to do this in zScript, but you can do it in other languages. The LuaSocket library for Lua, for example, will return you the code of any webpage, which you can then run a pattern search through. There's a thread on the forums that uses, if I recall, VBScript to return the TopMudSites webpage and do some stuff to it, which you might want to take a look at.

A potential implementation of LuaSocket would look a bit like this:

Code:
<trigger type="Alarm" language="Lua">
  <pattern>300</pattern>
  <value>if not http then http = require("socket.http") end

local page = http.request("www.whatever.com")
local players = string.match(page,"Players online: (%d+)")
zs.print("Players online: " .. players)</value>
</trigger>
_________________
Rorso's syntax colouriser.

- Happy bunny is happy! (1/25)
Reply with quote
Taz
GURU


Joined: 28 Sep 2000
Posts: 1395
Location: United Kingdom

PostPosted: Tue Mar 03, 2009 11:30 am   
 
That is waaayyy simpler than VBScript so don't bother looking at that particular post, also since Lua is native to CMUD I'm assuming it will also be a lot faster.
_________________
Taz :)
Reply with quote
TesterOfLimitz
Novice


Joined: 02 Jun 2007
Posts: 37

PostPosted: Wed Mar 04, 2009 11:50 am   
 
I went with the LuaForge route, and after a lil help from Fang, its working great :D

Ran into an issue with another website I was messing with... I'd like to beable to connect to samuraioflegend.net and download my inbox to cmud, but the site requires you be logged in (of course) to check it out. The problem lies with the fact that in order to refresh a page, it looks for a specific cookie, and if it can't be found, returns you to the sign in page.

Any ideas?
Reply with quote
Fang Xianfu
GURU


Joined: 26 Jan 2004
Posts: 5155
Location: United Kingdom

PostPosted: Wed Mar 04, 2009 12:01 pm   
 
Cookie data is actually appended to the calls the client makes of the server somehow. I was thinking about this after you PMed me about it and realised that one of my rss feeds requires that I send cookie data along - my rss reader simply has a "cookie data" string that it sends... somehow. It looks to me a lot like a POST method.

So basically, you need to read up on how cookie data is sent. LuaSocket will be able to do it, I'm sure, it's just a matter of formatting your query correctly. The nature of cookie data is that it doesn't change (because otherwise that defies the point of having them) so you'll be able to use a cookie viewer to find out what data the server is after from your browser - there're addons for firefox that make this very simple, for example. Then it's just a case of sending that data along with your query in whatever way the server wants.
_________________
Rorso's syntax colouriser.

- Happy bunny is happy! (1/25)
Reply with quote
TesterOfLimitz
Novice


Joined: 02 Jun 2007
Posts: 37

PostPosted: Wed Mar 04, 2009 12:11 pm   
 
Excellent! Thanks as always mate :D
Reply with quote
adamandkate
Wanderer


Joined: 14 Oct 2009
Posts: 57

PostPosted: Fri Jan 08, 2010 8:35 am   
 
are you able to explain a little to a noob about how this works. im placing it inside an alias or a trigger and it sits there doin nothing.

am just learning lua for the first time
Reply with quote
Taz
GURU


Joined: 28 Sep 2000
Posts: 1395
Location: United Kingdom

PostPosted: Fri Jan 08, 2010 2:09 pm   
 
Well what Fang has above is a 300 second alarm with it's scripting language set to lua, so every 5 minutes the lua code is run.

The lua code checks to see if the http object exists and if it doesn't then initialises it. If you haven't downloaded and put the LuaSocket library in the right place then it will never create that object.

Once the object is successfully created then it goes on to make a request from a web page and stores the result in a variable or possibly a table, not sure never used Lua before, called page. It then does some string matching on the result in page using a regex and stores that in players. It then outputs the contents of players onto your CMUD screen having concatenated "Players online: " to the front of it.

Is there anything in particular that you are wanting to achieve or was it just for learning's sake?
_________________
Taz :)
Reply with quote
adamandkate
Wanderer


Joined: 14 Oct 2009
Posts: 57

PostPosted: Fri Jan 08, 2010 3:22 pm   
 
ah.. ill get hold of the socket and try then .thanks for that

its purely for the sake of learning as it looked realllly interesting

thanks!
Reply with quote
ixy
Novice


Joined: 18 Mar 2007
Posts: 39

PostPosted: Fri Jan 08, 2010 8:13 pm   
 
The topic i found for socket help is this:

http://forums.zuggsoft.com/forums/viewtopic.php?t=32901

A nice guide :x
Reply with quote
Display posts from previous:   
Post new topic   Reply to topic     Home » Forums » CMUD General Discussion All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

© 2009 Zugg Software. Hosted by Wolfpaw.net