On demand objects
While writing a REST API it dawned on me I was going to need multiple database connections to multiple servers (a cluster isn’t really viable at this time…); the generic way for database objects is to use included files with the database object instantiated… well this from my view is additional overhead / processing time when you may not even use that database connection.
What i’ve devised is a basic singleton class which builds a object list of the objects to be instantiated BUT only a blueprint of the object so it doesn’t instantiate on load and ONLY instantiates when required.
It’s made up of two objects, one is a simple wrapper for the object to be created, the other, the main driver for doing so.
The wrapper is extremely basic, one constructor, one getter:
class demandObject { private $obj; private $params; private $funcName; function __construct($obj, $params, $funcName = false) { $this->name = (string) $obj; if($params === false) $params = array(); $this->params = $params; if($funcName === true) $funcName = 'getInstance'; $this->funcName = $funcName; } public function __get($var) { if(isset($this->$var)) return $this->$var; } }
Pretty self explanatory… the driver is a little more indepth:
class ondemand { public static $instance; private $pre = array(); private $post = array(); // Could have caching initiated here? function __construct() { self::$instance = $this; } public static function getInstance() { if(!isset(self::$instance)) new self; return self::$instance; } public function __get($ident) { return $this->__initObject($this->pre[$ident]); } public function __set($ident, demandObject $obj) { $this->pre[$ident] = $obj; } private function __initObject(demandObject $obj) { if(isset($this->post[$obj->name])) return $this->post[$obj->name]; if($obj->funcName === false) { $ro = new ReflectionClass($obj->name); $o = $ro->newInstanceArgs($obj->params); } else { $o = call_user_func_array(array($obj->name, $obj->funcName), $obj->params); } $this->post[$obj->name] = $o; return $o; } }
What we have here is a singleton pattern being used to maintain a single array of objects to be instantiated, depending if this object has a primary function to call first, or whether it just calls the object’s constructor, we instantiate the object in a getter using this logic.
I’m not going to really go into much detail, here’s an example of how it would be used:
// example object class x { private $myX; function __construct($x) { $this->myX = $x; } public function getX() { return $this->myX; } } $demand = ondemand::getInstance(); $demand->myX = new demandObject('x', array('Hello World')); /* Script that doesn't require the object x... */ // aha, now we need to use it! echo $demand->myX->getX(), "\n";
So as you can see, x isn’t instantiated unless needed, obviously not really required for an object of such simplicity as x, but it serves as an example.
If the object also used a singleton pattern, and its method was called ‘getInstance’ you can simply tell provide it a boolean of true and it’ll use that rather than the default constructor:
$demand->myY = new demandObject('y', array('Hello World'), true);
It can also be provided as a string to the method name. Simple but elegant.
No commentsmrsync – rsync queuing script
I have an upcoming task to transfer a LOT of files from one server to another, I wrote this to try and speed up the transfer process at the sake of IO load.
While writing this script a single rsync process has been running, it would take on estimate 10 days to complete, this is even with the latest rsync which has had huge improvements!
It’s still a little rough around the edges, but for my use it should cope, later today I’ll add in support for it to only start a process if the the same rsync isn’t running, if it is, log it as a running job and have it handle the same way as any other running job.
For help on using it, run it without any arguments or with ‘–help’ flag.
#!/bin/bash #!/bin/bash # # @author Ed Cradock # @date 10 February 2010 # # Mrsync - Execute multiple rsync's with queue system # # Flags NEXTARG=0 ERROR=0 GOTO_DAEMON_MODE=0 # User set options FILE_LIST=0 LOG_DIR="./mrsync-logs" OP_DIR="$HOME/.mrsync" JOBS=4 IS_SSH="" # Counters RUNNING_RSYNC=0 TOTAL_JOB_COUNT=0 CURRENT_JOB=0 # Temporary file names TMP_RUNNING_JOBS=".mrsync-running-jobs" TMP_COMPLETED=".mrsync-completed" function PROCESS_BY_ID_EXISTS { if [[ $1 != [0-9]* ]]; then return 0 fi if [ $(ps --pid $1 --no-header | wc -l) -gt 0 ]; then return 1 else return 0 fi } # Some what flimsy... # but let's ensure there's no rsync running doing the same task function PROCESS_BY_ARG_EXISTS { return $(ps aux --no-header | awk '{ print $11 }' | grep -E "^rsync" | grep $1 | wc -l) } function PROCESS_COMPLETED { while read complete; do COMPLETE_PID=${complete%% *} COMPLETE_PS=${complete#* } if [ "$1" == "$COMPLETE_PID" -a "$2" == "$COMPLETE_PS" ]; then return 1 fi done < $TMP_COMPLETED return 0 } function FLUSH_TEMPORARY_FILES { :> $TMP_RUNNING_JOBS :> $TMP_COMPLETED } function DO_JOB { CURRENT_JOB=$(expr $CURRENT_JOB + 1) RSYNC="rsync -avz $IS_SSH $1 $2" RUNNING_RSYNC=$(expr $RUNNING_RSYNC + 1) CLEAN_NAME=$(echo $RSYNC | sed -r 's/[^a-zA-Z0-9]+/-/g') ($RSYNC &> $LOG_DIR/$CLEAN_NAME) & JOB_PID=$! echo "$JOB_PID $RSYNC" >> $TMP_RUNNING_JOBS echo "[Executing] $RSYNC" } function DAEMON_MODE { JOBS_COMPLETE=0 while [ 1 ]; do while read entry; do CIT=$(expr $CIT + 1) ENTRY_PID=${entry%% *} ENTRY_PS=${entry#* } PROCESS_BY_ID_EXISTS $ENTRY_PID if [ $? == 0 ]; then PROCESS_COMPLETED $ENTRY_PID "$ENTRY_PS" if [ $? == 0 ]; then echo "[Completed] $ENTRY_PS" echo "$ENTRY_PID $ENTRY_PS" >> $TMP_COMPLETED if [ $TOTAL_JOB_COUNT -eq $CURRENT_JOB -a $CIT -eq $TOTAL_JOB_COUNT ]; then JOBS_COMPLETE=1 break fi RUNNING_RSYNC=$(expr $RUNNING_RSYNC - 1) JOBS_REMAINING=$(expr $TOTAL_JOB_COUNT - $CURRENT_JOB) if [ $JOBS_REMAINING -gt 0 ]; then i=0 while read line; do i=$(expr $i + 1) SRC=${line%% *} DEST=${line#* } if [ $i -gt $CURRENT_JOB -a $RUNNING_RSYNC -lt $JOBS ]; then DO_JOB "$SRC" "$DEST" fi done < $FILE_LIST fi fi fi if [ $CIT -eq $TOTAL_JOB_COUNT ]; then CIT=0 fi done < $TMP_RUNNING_JOBS if [ $JOBS_COMPLETE -eq 1 ]; then echo "All done! :)" exit fi done } function HELP { echo -e " \033[1mmrsync - Queued rsync'ing\033[0m" echo " --file-list <filename> - mandatory, a file to read in the required rsyncs." echo " --log-dir <filename> - specify log directory [ mrsync-logs ]" echo " --jobs <max concurrent> - maximum jobs to execute at one time [4]" echo " --op-dir <filename> - specify an alternate mrsync operations directory [~/.mrsync]" echo " --ssh - to use ssh as transport medium" echo " --help - display this message" exit } # Handle user set options for arg in $*; do # Handle argument values if [ "$NEXTARG" != 0 ]; then case $NEXTARG in FILE_LIST) if [ ! -e $arg ]; then echo "Error: Filename list file does not exist." ERROR=1 fi ;; JOBS) if [[ $arg != [0-9]* ]]; then echo "Error: Jobs must be an integer, defaulting to $JOBS." ERROR=1 fi ;; LOG_DIR) if [ ! -d $arg ]; then mkdir $arg &> /dev/null if [ ! -d $arg ]; then echo "Log directory does not exist and an attempt to create it failed." ERROR=1 fi fi ;; OP_DIR) if [ ! -d $arg ]; then mkdir $arg &> /dev/null if [ ! -d $arg ]; then echo "Log directory does not exist and an attempt to create it failed." ERROR=1 fi fi ;; *) ERROR=1 ;; esac if [ $ERROR == 1 ]; then NEXTARG=0 continue fi read $NEXTARG <<< $arg NEXTARG=0 fi if [ ${arg:0:2} == "--" ]; then case ${arg:2} in file-list) NEXTARG="FILE_LIST";; jobs) NEXTARG="JOBS";; log-dir) NEXTARG="LOG_DIR";; op-dir) NEXTARG="OP_DIR";; ssh) IS_SSH="-a ssh";; help) HELP;; *) echo "unknown argument $arg" help ;; esac fi done if [ ! -e $FILE_LIST ]; then HELP fi if [ ! -e $LOG_DIR ]; then mkdir $LOG_DIR &> /dev/null if [ ! -e $LOG_DIR ]; then echo "Failed to create log dir, please specify one with --log-dir or give permissions for mrsync to create one within this directory." exit fi fi if [ ${LOG_DIR#${LOG_DIR%?}} == "/" ]; then LOG_DIR=$(echo $LOG_DIR | sed 's/.$//') fi if [ ! -d $OP_DIR ]; then mkdir $OP_DIR &> /dev/null if [ ! -d $OP_DIR ]; then echo "Failed to create temporary directory, please specify one with --op-dir or give permissions for mrsync to create one at $OP_DIR." exit fi fi if [ ${OP_DIR#${OP_DIR%?}} == "/" ]; then OP_DIR=$(echo $OP_DIR | sed 's/.$//') fi TMP_RUNNING_JOBS="$OP_DIR/$TMP_RUNNING_JOBS" TMP_COMPLETED="$OP_DIR/$TMP_COMPLETED" FLUSH_TEMPORARY_FILES while read line; do SRC=${line%% *} DEST=${line#* } TOTAL_JOB_COUNT=$(expr $TOTAL_JOB_COUNT + 1) PROCESS_BY_ARG_EXISTS $SRC $DEST $args if [ $? == 0 ]; then if [ $RUNNING_RSYNC -lt $JOBS ]; then DO_JOB "$SRC" "$DEST" else echo "Dropping into daemon mode" break fi fi done < $FILE_LIST DAEMON_MODE
Lazy way for adding new IP addresses on Debian
Today I was given the fun task of adding around 30 IP addresses to a server, well being the lazy person I am, I decided to script it… here’s what I came up with:
#!/bin/bash ## # Add separate virtual interfaces for a list of of IP addresses # to the debian specific /etc/network/interfaces # @author Ed Cradock # ## NETWORK_IFS="/etc/network/interfaces"; CURRENT_IFNO=`grep "iface eth0" $NETWORK_IFS | tail -n 1 | cut -d":" -f2 | cut -d" " -f1`; CURRENT_NETM=`grep "netmask" $NETWORK_IFS | head -n 1 | cut -d" " -f2`; function addInterface { CURRENT_IFNO=$(($2 + 1)); echo -e "\nauto eth0:$CURRENT_IFNO" >> $1; echo -e "\niface eth0:$CURRENT_IFNO inet static" >> $1; echo -e "\taddress $3" >> $1; echo -e "\tnetmask $4" >> $1; # Finally bring up the interface ifup "eth0:$CURRENT_IFNO"; } if [ $# -ne 0 ]; then if [ -f $1 ]; then for IPADDR in `cat $1`; do addInterface $NETWORK_IFS $CURRENT_IFNO $IPADDR $CURRENT_NETM; done else for IPADDR in $*; do addInterface $NETWORK_IFS $CURRENT_IFNO $IPADDR $CURRENT_NETM; done fi else echo "Usage: $0 [<file with ipaddresses>|<ipaddress>, ...]"; fi
Sumarised, it takes either a file full of IP addresses (this is the technique I used) and adds the interfaces to /etc/network/interfaces, then brings the interface up; Alternatively it can accept multiple ip addresses given as arguments.
Definitely a script I’ll use in the future.
No commentsrsync woes
I have been using rsync recently to transfer a farm of files from one server to another, I managed to make a glaring omission when reading the documentation which really left me in a rut…
Summed up, rsync … /src /destination is NOT rsync /src/ destination, by missing the trailing ‘/’ at the end of the source, rsync will copy a directory named ’src’ into destination directory; however WITH it, it copies the files in src to destination (the desired behaviour).
This has caused me a lot of issues, so watch out!
No commentsUsing mysqldump for a set amount of results
My friend Greg has kindly been developing a script based on a MySQL database table I have created, the table at current is just shy of 50,000 rows and he needed a sampler of the rows to work on…
Unfortunately mysqldump doesn’t have a switch to specify a set number of rows, but it is possible to do via the ‘–where’ switch, like so:
mysqldump -u mydbuser -p --where="1=1 LIMIT 10" mydb mytable
Unfortunately you can’t just set the where clause to ‘LIMIT 10′ as that creates an error, but have it force an always true condition and you’re away.
A useful little trick.
No commentsMSN on the move
Okay been a while since I posted anything, just been rather busy.
So my phone runs Windows Mobile 6 and because it’s a HTC handset released by Orange the Microsoft MSN Messenger which usually accompanies Windows Mobile 6 is not present.
I’ve been on the lookout for a free alternative and I stumbled across a nice client today named ‘Palringo’, what makes Palringo different is that it does not operate from your phone directly, you have to create a Palringo account; then add the accompanying services, such as MSN etc.
These added services are then stored on your account, the security of this concerns me as my password is stored somewhere on their server. I hope they use some form of key based hashing, the key being the password you enter to get into the account, then if their system is ever compromised all they then have is a bunch of hashed passwords.
I signed up for a new account and linked it to an MSN Live account, what’s pretty nifty about this software though is the ability to send voice messages and photo’s directly from the handset, you click ’send photo’ and it loads your phone’s camera, you take the shot, and when you get back to Palringo, the image has sent, very useful.
Very nice looking app too, here’s a screenshot I pinched from the website:
Supports lots of platforms too, I’m just not quite sure what the catch is…
1 commentSmallest PHP IRC bot
A friendly one day competition was held on Zymic’s IRC a few weeks ago where the challenge was to write the smallest (in characters and lines) IRC bot, here was my entry:
1 line; 253 characters
Highly unoptimised, but that was to keep the character count down.
Really fun challenge, I did end up winning, but it was definitely the fun of writing it more than the winning.
Some of the other entries can be found here : http://competition.pastebin.com
No commentsRestricting outgoing urls with iptables
I was having a nose through the iptables manual and came across the extension ‘owner’, basically, here’s the manual page excerpt:
owner This module attempts to match various characteristics of the packet creator, for locally-generated packets. It is only valid in the OUTPUT chain, and even this some packets (such as ICMP ping responses) may have no owner, and hence never match. --uid-owner userid Matches if the packet was created by a process with the given effective user id. --gid-owner groupid Matches if the packet was created by a process with the given effective group id. --pid-owner processid Matches if the packet was created by a process with the given process id. (Please note: This option requires kernel support that might not be available in official Linux kernel sources or Debian’s packaged Linux kernel sources. And if support for this option is available for the specific Linux kernel source ver‐ sion, that support might not be enabled in the current Linux kernel binary.) --sid-owner sessionid Matches if the packet was created by a process in the given session group. (Please note: This option requires kernel support that might not be available in official Linux kernel sources or Debian’s packaged Linux kernel sources. And if support for this option is available for the specific Linux kernel source ver‐ sion, that support might not be enabled in the current Linux kernel binary.) --cmd-owner name Matches if the packet was created by a process with the given command name. (Please note: This option requires kernel support that might not be available in official Linux kernel sources or Debian’s packaged Linux kernel sources. And if support for this option is available for the specific Linux kernel source ver‐ sion, that support might not be enabled in the current Linux kernel binary.) NOTE: pid, sid and command matching are broken on SMP
It then dawned on me what I could use this for.
I wanted a way to restrict outgoing urls used within php, so for example, script-x could communicate with api.recaptcha.net, but script-x could not communicate say with mrmiagisguidetokarate.com.
First I created a new chain within the filter table:
~$ iptables -N outgoingurls
Then I set the default policy for this chain to ‘ACCEPT’
iptables -A outgoingurls -j ACCEPT
We now add the iptables rule which will drop all outgoing connections from apache:
iptables -A OUTPUT -m owner --uid-owner www-data -j REJECT
Notice the REJECT as opposed to DROP, REJECT sends an ICMP response back to inform that the connection is refused and does not hang as much.
Now to allow just recaptcha.net (64.34.251.150):
iptables -A OUTPUT -p tcp -m owner --uid-owner www-data -d 64.34.251.150 -j outgoingurls
If we were ever wanting to disable all urls we can set the default policy of the outgoingurls chain to REJECT, or else we can just flush the chain.
The caveat to this technique that it can only really be done on an IP address level, that IP address may host multiple sites which could lead to nasties not being filtered, so user beware.
An optional technique but using the owner extension again is to redirect outgoing urls to an intermediary proxy which could sift through the payload and filter it based on that; the downside to this technique is that although it does the job it’s a very long winded and expensive (resources wise) process, but if you were to want to something similar, you could filter to your proxy like so:
iptables -t nat -A PREROUTING -t tcp -m owner --owner-uid www-data -j DNAT --to 192.168.1.100:8080 iptables -t nat -A OUTPUT -t tcp -m owner --owner-uid www-data -j DNAT --to 192.168.1.100:8080 iptables -t nat -A PREROUTING -t tcp -m owner --owner-uid www-data -s 192.168.1.100 -j ACCEPT iptables -t nat -A OUTPUT -t tcp -m owner --owner-uid www-data -s -s 192.168.1.100 -j ACCEPT
You would need to bind your proxy to 192.168.1.100, so you could create an additional address to bind onto like so:
ifconfig eth0:0 192.168.1.100 netmask 255.255.255.0
That done anything outgoing (that is not from your proxy address) will be passed off the proxy, anything that is from your proxy is allowed.
I wrote this in a bit of a rush, so apologies for any inaccuracies, I’ll review it next time I’m online.
1 commentObfuscation… it’s like cling film over the toilet seat.
A friend of mine started telling me about a php script obfuscation / encoding application, I was none too convinced that it could have any viable use to thwart the theft of code, and my preconceptions were bang on the nail.
I shall from this point on be calling the application ‘obfuscator-x’(so not to jeopardise anyone’s codebase who actually attempt to protect their source with the product).
Obfuscation-x basically is a windows application which takes your source files then garbles them to the extent they are not easily readable, yet can be used as you normally would. Obfuscation-x does this through the means of shifting bytes backwards and forwards in a predetermined algorithm.
Let me show you the source of the original script which is being encoded / obfuscated:
<?php echo 'Hello World'; ?>
Now we look at the obfuscated / encoded source:
if (!function_exists("a")) { function a($b) { $b = base64_decode($b); $a = 0; $c = 0; $d = 0; $e = (ord($b[1]) << 8) + ord($b[2]); $f = 3; $g = 0; $h = 16; $i = ""; $j = strlen($b); $k = __FILE__; $k = file_get_contents($k); $l = 0; preg_match(base64_decode("LyhwcmludHxzcHJpbnR8ZWNobykv"), $k, $l); for (;$f<$j;) { if (count($l)) exit; if ($h == 0) { $e = (ord($b[$f++]) << 8); $e += ord($b[$f++]); $h = 16; } if ($e & 0x8000) { $a = (ord($b[$f++]) << 4); $a += (ord($b[$f]) >> 4); if ($a) { $c = (ord($b[$f++]) & 0x0F) + 3; for ($d = 0; $d < $c; $d++) $i[$g+$d] = $i[$g-$a+$d]; $g += $c; } else { $c = (ord($b[$f++]) << 8); $c += ord($b[$f++]) + 16; for ($d = 0; $d < $c; $i[$g+$d++] = $b[$f]); $f++; $g += $c; } } else $i[$g++] = $b[$f++]; $e <<= 1; $h--; if ($f == $j) { $k = implode("", $i); $k = "?".">".$k."<"."?"; return $k; } } } } eval(a("QAAAPD9waHAgZWNobyAnSGVsbAAEbyBXb3JsZCc7ID8+IAAU"));
Variables and function names have been changed
The first thing that initially hit me is the amount of extra bytes your file needs just to be interpreted…
If you’re not experienced in php you might think that it looks pretty secure, but how wrong you’d be, first let’s look at that last line:
eval(a("QAAAPD9waHAgZWNobyAnSGVsbAAEbyBXb3JsZCc7ID8+IAAU"));
This is the encoded source in a base64 format but being passed through the ‘a’ function to decode it… so all we have to do is output the bit going into eval, now the developers of the software have attempted to prevent this, but in such a way it’s almost laughable, where, here’s where:
preg_match(base64_decode("LyhwcmludHxzcHJpbnR8ZWNobykv"), $k, $l); for (;$f<$j;) { if (count($l)) exit;
Basically this decoded says this:
/(print|sprint|echo)/
Now that is damn right ridiculous, that doesn’t even take into account ‘var_dump’, ‘var_export’, now of course it’s impossible to have a 100% fool proof obfuscator but steps like this if they are attempting to thwart attempts should most definitely be taken into consideration. If you’ve followed thus far then it’s pretty obvious how to get the source:
var_dump(a("QAAAPD9waHAgZWNobyAnSGVsbAAEbyBXb3JsZCc7ID8+IAAU"));
It’s hardly going to take a genius to work that out!
So it got me thinking… could you obfuscate the function’s being called, well the answer to an extent (and at the price of inefficiency!) is yes. This is really just a test case and I personally wouldn’t recommend even thinking about using a technique like this…
So here is my attempt at function name obfuscation:
$a=array(0x69,0x6d,0x70,0x6c,0x6f,0x64,0x65); array_unshift($a, str_repeat('%c',count($a))); $f=call_user_func_array('sprintf', $a); print_r($f('|', array('a','b','c')));
Broken down the first line is an array of hexadecimal representations of the character codes, second line places the proper character formatting string within the first entry in the array (so it’s the first argument passed to sprintf), the 3rd line is pretty obvious and assigns the function name to $f, then calling $f( … ) gives you the function. Ridiculously impractical, but then again, so are most obfuscators.
To build any function in this way I wrote a quick function:
The array string could then be used within the script, you’ll have to excuse the short, non-concise variables names, it is after all a very dirty function.
Overall, don’t buy obfuscators… if you really want a better chance of protecting your source you can use something like Zend Encoder or IonCube, these have the distinct advantage that they compile your script into byte code, still not impossible to reverse engineer, but you’re more likely to keep the majority of snoopers out (until some bright spark releases an application to the masses for decoding it).
I believe that about concludes this blog post.
No commentsFirst of many…
I’m not too seasoned at writing blogs, so bear with me…
This blog is really just a place for me to document my little findings, the topics will tend to be related to Linux, server administration and web development but I’m sure I’ll digress at points and blab on about some other tuneful topic.
I will start off by saying, I will not be one of those bloggers who feels like documenting his entire life… if you want to know what I had for dinner, what I got up to, or perhaps what I’m wearing, or even the current track I’m listening to then: pass me your name, address and I’ll endeavour to take out a restraining order on you. 
I’ll also admit this is more-so a persistent repertoire for my findings / musings than contributions to help the masses… but if it helps, all the better.
I also feel obliged to almost apologise for using 3rd party software, I do consider myself a bit of a purist, but I needed a blog quick to jot down my findings and with more pressing things to work on I really didn’t have the time to write my own blog… maybe in the future.
Now that’s out the way, I’m off to find a plugin for syntax highIighting within my posts and after that I suppose I better get writing some meaningful content!
No comments