• Welcome to Computer Association of SIUE - Forums.
 

Open Source Projects

Started by Jarod Luebbert, 2007-08-29T15:02:18-05:00 (Wednesday)

Previous topic - Next topic

Jarod Luebbert

Does CAOS or anyone else on campus ever put together any open source projects? I would really like to either start one or be involved in one. I think it could benefit everyone.
Jarod Luebbert
Computer Science Major

William Grim

Maybe you could help test an anonymizing proxy I'm starting soon.  Also, if I can implement a Solaris version of ltrace, you might be able to help there too; first I need to see if the Sparc chips support some circuitry I will need to implement ltrace.
William Grim
IT Associate, Morgan Stanley

Kit

SIUe Computer Science Graduate

my_handle

would this be similar to privoxy?  Possibly combined with something like tor?

bandyguy

I'm not sure what any of that is, but working on an open source project would be cool. My programing skills are next to nothing (taking CS150) so I probably wouldn't be much help but I would try.

William Grim

It's more like Tor itself but at the IP level rather than the TCP/IP level.  There are a lot of difficulties that will have to be overcome.  First, I need to determine the feasibility and expected usefulness of the application.  I believe it's possible to create anonymization at the IP level via encapsulation or something else, but it doesn't matter if it's not going to be useful to even a handful of people.  However, when I determine all this, I will give more details.
William Grim
IT Associate, Morgan Stanley

William Grim

Speaking of ltrace being ported to Solaris, I just decided not to do that.  After reading more on dtrace, it appears dtrace is already much more advanced.  I had heard how good it was but hadn't taken a look yet since I haven't used Sun machines much until now, and I can only see my Sun Solaris use increasing greatly in the future.
William Grim
IT Associate, Morgan Stanley

Jarod Luebbert

I'm not very experienced either. I have been programming in C++ for around a year and I'm currently in CS-140. Even if the programs were useless it would still increase everyones programming skill.
Jarod Luebbert
Computer Science Major

bandyguy

any Idea what kind of project? I think google has a ton of unfinished open source project too. I always have trouble thinking of ideas for programming, cause most of the stuff I would like to program is beyond my level.

Kit

I'd recommend starting a project that benefits mostly yourself (for your first couple projects anyways). Not only does it help to practice and learn more programming, but then you end up with a program that at least you will find useful.

I know that I've written a simple program to generate the html for the CAB events page. In fact, I would upgrade it so that other people besides me would be able to use it, but I don't have time.

By the time you finish a couple, I think it will be easier to find a group project to join (as well as becoming more experienced and confident in your ability).
SIUe Computer Science Graduate

Shaun Martin

Quoteamphoterous wrote:
http://caos.cs.siue.edu/uploads/smil3f1fff3820aa9.gif" alt="" />

In b4 Shaun.

 :cry:
Shaun Martin
SIUE Alumni
Associate IT Analyst, AT&T Services, Inc. St. Louis, MO.

Shaun Martin

Quoteamphoterous wrote:
:huh:

In b4 Shaun.

 :cry:

My emoticons are broken!
Shaun Martin
SIUE Alumni
Associate IT Analyst, AT&T Services, Inc. St. Louis, MO.

William Grim

Here is some code that I'd like anyone to try and debug.  It's called "ExProxy" and only requires Python 2.4+.  I've been testing it on OS X but am fairly sure it should work on Windows and am confident it will run on other Unix platforms.

The idea behind the proxy is pretty basic.  It does the normal proxy thing for most URIs; however, if you pass special URI strings to the proxy, it will execute custom code and return the results to the browser.  Right now it only has a basic special command: 'g'.  You can do "g blah blah", and it will forward "blah blah" to google and return the results.

Please keep in mind that I'm having some issues getting this to work with gmail and other similar sites right now.  It does work with CAOS and facebook at the moment.  Also, it is not yet multithreaded; I've been putting my efforts into making it conform to RFCs first.  This means that it's not very fast, and if it blocks on something, you will need to hit ^C once to cancel the request if you no longer wish to wait.  All this will be properly handled later.

And without further adieu, here is the code that everyone can test and (hopefully) debug!


#!/usr/bin/env python

import urllib as u1
import urllib2 as u2
import urlparse as p
import CGIHTTPServer as cgi
import BaseHTTPServer as b
import random as r
import cookielib as cl
import sys
import socket

_debug = False

class ExHTTPRequestHandler(cgi.CGIHTTPRequestHandler, object):
def _handleGoogle(self, data):
url = 'http://www.google.com/'
if data is not None:
url += 'search?hl=en&safe=off&q=' + '+'.join(data)
return 200, self.headers, self._makeCall(url)

def _handleCAOS(self, data):
"""
XXX: Not yet implemented.
"""
if _debug:
print "In the CAOS function."
sys.stdout.flush()
url = 'http://caos.siue.edu/'
if data is not None:
url += None

keys = {'g':_handleGoogle, 'c':_handleCAOS}
_opener = u2.build_opener()
#protocol_version = "HTTP/1.1"

def _makeCall(self, url, params=None):
try:
return self._makeCall2(url, params)
except u2.HTTPError, e:
response, headers, output = e.code, e.hdrs, e.fp.read()
e.fp.close()
if _debug:
print >>sys.stderr, "ERROR RESPONSE:", response
print >>sys.stderr, "ERROR HEADERS:"
print >>sys.stderr, headers
print >>sys.stderr, e
return response, headers, output

def _makeCall2(self, url, params):
if _debug:
print self.command + " " + url + "."
print "HEADERS FROM CLIENT:"
print self.headers
sys.stdout.flush()

"""
if 'max-forwards' in self.headers:
RFC 2616 requires us to check this field if it exists.
left = int(self.headers['max-forwards'])
if left <= 0:
return 200, self.headers, ""
"""

o = self._opener
o.addheaders = []
for header in self.headers:
"""
Forward most headers but ignore some others
for now (RFC 2616).
"""
if header not in ['host', 'connection', 'keep-alive',
 'proxy-authenticate',
 'proxy-authorization',
 'te', 'trailers',
 'transfer-encoding', 'upgrade']:
o.addheaders.append((header,
    self.headers[header]))

# Required by RFC 2616.
via = self.request_version + ' ' + socket.gethostname()
if 'via' in self.headers:
o.addheaders.append(('via',
   self.headers['via'] + ', ' + via))
else:
o.addheaders.append(('via', via))

if params is not None and _debug:
print "POST DATA: " + params
sys.stdout.flush()
data = o.open(url, params)
if _debug:
print "HEADERS FROM SERVER:"
sys.stdout.flush()
h = data.info()
if 'via' in h:
headers = [('via', h['via'] + ', ' + via)]
del h['via']
else:
headers = [('via', via)]
for header in h:
headers.append((header, h[header]))

if 'content-length' in headers:
output = data.read(int(headers['content-length']))
else:
output = data.read()
headers.append(('content-length', str(len(output))))

if _debug:
for (key, value) in headers:
print key + ': ' + value
sys.stdout.flush()

data.close()
return 200, headers, output

def _sendResponse(self, response, headers, output):
self.send_response(response)
for header in headers:
self.send_header(header[0], header[1])
self.end_headers()
self.wfile.write(output)

def do_POST(self):
inputLen = int(self.headers['content-length'])
input = self.rfile.read(inputLen)
response, headers, output = self._makeCall(self.path, input)
self._sendResponse(response, headers, output)

def do_GET(self):
components = p.urlparse(self.path)
host = u2.unquote(components[1])
params = host.split(' ')
if params[0] in self.keys:
if len(params) > 1:
key = params[0]
params = params[1:]
else:
params = None
input = params
handler = lambda params: self.keys[key](self, params)
else:
input = self.path
handler = lambda params: self._makeCall(params)

response, headers, output = handler(input)
self._sendResponse(response, headers, output)

def run(serverAddress, serverClass=b.HTTPServer,
handlerClass=ExHTTPRequestHandler):
print("Now serving requests at %r:%r." %
     (serverAddress[0], serverAddress[1]))
sys.stdout.flush()
httpd = serverClass(serverAddress, handlerClass)
httpd.serve_forever()

if '__main__' == __name__:
_debug = True
run(('', 8080))
William Grim
IT Associate, Morgan Stanley

William Grim

Hello again.

I'm going to post up here one more time with my proxy code.  After that, I'll start getting it placed on my web server for download when I get a little project page up for it.

So, here is a proxy that, as far as I know, properly conforms to most standards.  It still doesn't work with gmail and other sites that use XmlHttpRequests, but I'll be working on that soon (i.e. next weekend when I'm not working).

This proxy only supports incoming HTTP/1.0 requests right now, but it makes outgoing HTTP/1.1 requests.  Mainly, this means that the proxy isn't as fast as it can be (not to mention it's not using select(2) yet).

It should be pretty easy to get it to do HTTP/1.1, seeing as how the library I'm using supports it, but I haven't bothered with that yet.  Look for that in a future release.

So, once again, here is the updated code!

P.S.: I posted this code using my proxy just to show it works :)


#!/usr/bin/env python

import urllib as u1
import urllib2 as u2
import urlparse as p
import CGIHTTPServer as cgi
import BaseHTTPServer as b
import random as r
import cookielib as cl
import sys
import socket
import inspect

_debug = False

class ExHTTPRequestHandler(cgi.CGIHTTPRequestHandler, object):
def _handleGoogle(self, data):
url = 'http://www.google.com/'
if data is not None:
url += 'search?hl=en&safe=off&q=' + '+'.join(data)
return 200, self.headers, self._gatherDataWrapper(url)

def _handleCAOS(self, data):
"""
XXX: Not yet implemented.
"""
if _debug:
print "In the CAOS function."
sys.stdout.flush()
url = 'http://caos.siue.edu/'
if data is not None:
url += None

keys = {'g':_handleGoogle, 'c':_handleCAOS}
_opener = u2.build_opener()
#protocol_version = "HTTP/1.1"

# XXX: Will likely be good to turn this into a dict later, using
# socket file numbers as keys so we can start multiplexing our
# data streams.
_headerBuffer = ""

def send_header(self, key, value):
caller = inspect.stack()[1][3]
if 'send_response' == caller:
"""
Ignore this caller.  It attempts to send its own
'Server' and 'Date' headers which breaks RFC 2616.
RFC 2616 states that 'Server' and 'Date' fields are
end-to-end fields, meaning we should simply forward
the ones we get from the origin server.
"""
return
self._headerBuffer += key + ": " + value + "\r\n"

def end_headers(self):
self.wfile.write(self._headerBuffer)
self.wfile.write("\r\n")
self._headerBuffer = ""

def _gatherDataWrapper(self, url, params=None):
try:
return self._gatherData(url, params)
except u2.HTTPError, e:
response, headers, output = e.code, e.hdrs, e.fp.read()
e.fp.close()
if _debug:
print >>sys.stderr, "ERROR RESPONSE:", response
print >>sys.stderr, "ERROR HEADERS:"
print >>sys.stderr, headers
print >>sys.stderr, e
return response, headers, output

def _gatherData(self, url, params):
if _debug:
print self.command + " " + url + "."
print "HEADERS FROM CLIENT:"
print self.headers
sys.stdout.flush()

"""
if 'max-forwards' in self.headers:
RFC 2616 requires us to check this field if it exists.
left = int(self.headers['max-forwards'])
if left <= 0:
return 200, self.headers, ""
"""

o = self._opener
o.addheaders = []
for header in self.headers:
"""
Forward most headers but ignore some others
for now (RFC 2616).
"""
if header not in ['connection', 'keep-alive',
 'proxy-authenticate',
 'proxy-authorization',
 'te', 'trailers',
 'transfer-encoding', 'upgrade']:
o.addheaders.append((header,
    self.headers[header]))

# Required by RFC 2616.
via = self.request_version + ' ' + socket.gethostname()
if 'via' in self.headers:
o.addheaders.append(('via',
   self.headers['via'] + ', ' + via))
else:
o.addheaders.append(('via', via))

if params is not None and _debug:
print "POST DATA: " + params
sys.stdout.flush()
data = o.open(url, params)
if _debug:
print "HEADERS FROM SERVER:"
sys.stdout.flush()
h = data.info()
if 'via' in h:
headers = [('via', h['via'] + ', ' + via)]
del h['via']
else:
headers = [('via', via)]
for header in h:
headers.append((header, h[header]))

if 'content-length' in headers:
output = data.read(int(headers['content-length']))
else:
output = data.read()
headers.append(('content-length', str(len(output))))

if _debug:
for (key, value) in headers:
print key + ': ' + value
sys.stdout.flush()

data.close()
return 200, headers, output

def _sendResponse(self, response, headers, output):
self.send_response(response)
for header in headers:
self.send_header(header[0], header[1])
self.end_headers()
self.wfile.write(output)

def do_POST(self):
inputLen = int(self.headers['content-length'])
input = self.rfile.read(inputLen)
response, headers, output = self._gatherDataWrapper(self.path,
   input)
self._sendResponse(response, headers, output)

def do_GET(self):
components = p.urlparse(self.path)
host = u2.unquote(components[1])
params = host.split(' ')
if params[0] in self.keys:
if len(params) > 1:
key = params[0]
params = params[1:]
else:
params = None
input = params
handler = lambda params: self.keys[key](self, params)
else:
input = self.path
handler = lambda params: \
   self._gatherDataWrapper(params)

response, headers, output = handler(input)
self._sendResponse(response, headers, output)

def run(serverAddress, serverClass=b.HTTPServer,
handlerClass=ExHTTPRequestHandler):
print("Now serving requests at %r:%r." %
     (serverAddress[0], serverAddress[1]))
sys.stdout.flush()
httpd = serverClass(serverAddress, handlerClass)
httpd.serve_forever()

if '__main__' == __name__:
_debug = True
run(('', 8080))
William Grim
IT Associate, Morgan Stanley