blog.ARConrad.com

To improve performance on a current Ruby project I’m doing a lot of threading and ‘processing ahead’. However, part of my code cannot be threaded because the fork would inherit an instance variable that is not thread safe.

To overcome this I’m using fibers and in doing so I noticed that when you’re ‘processing ahead’ at the end of a loop and then resume the fiber at the start of the loop the local variables are not released by Ruby’s garbage collector. This allows for some fairly bizarre yet brilliant opportunities. If you run the following code you’ll see what I mean:

(download as a gist: https://gist.github.com/2641725)

5.times do

 #skips on first run since @fiber doesn't yet exist.
 @fiber.resume if defined? @fiber 

 @fiber = Fiber.new do

   puts "I'm running at end of loop and setting \
a local variable 'language'"

   language = "Ruby"

   Fiber.yield

   puts "I'm running at the start of the loop and \
I still know the local variable 'language' is \
'#{language}' from the last iteration of the loop! \
I.E. I was not garbage collected.\n\n"

 end

 @fiber.resume

 sleep 1
end

Dear fellow Quants, are you tired of MetaTrader’s clunky MQL4 language? Would you prefer to write forex trading algorithms in:

Ada, Bash, Basic, C, Chicken Scheme, Common Lisp, C#, C++, D, Erlang, F#, Flex, Go, Guile, Haskell, Haxe, Java, Javascript (flash), Lua, Node.js, Objective-C, Objective Caml, ooc, Perl, PHP, Python, Racket, R, REBOL 2, REBOL 3, Red, Ruby, Ruby (FFI), Scala, Smalltalk, Twisted (Python), etc?

The limitations of MetaTrader4′s MQL4 language really bother me and I don’t want to spend $$$ on forex API’s when I’m just doing research and development.

After thinking about this for a while, I decided to take a week and write “MQL4ZMQ“, a complete set of “C”-spec bindings for ZeroMQ v2.1 written in the MQL4 language. The use of ZeroMQ messages allows simplified backend communication with MetaTrader4 from any language with ZeroMQ bindings.

For more details, check out MQL4ZMQ.org or head directly over to my Github account to grab the code and start building your custom ZeroMQ-MetaTrader bridge.

IMPORTANT NOTE: You are free to use MQL4ZMQ in your trading projects as long as you agree that no warranty is provided whatsoever.

—————–-
now playing: Freestylers – Cracks (feat. Bella Humble) Flux Pavilion Remix

I’ve been including ActiveRecord for awhile simply to use a couple of it’s non-database functions so I decided it was time to write a couple simple snippets that I could use instead; saving my programs the dependency and memory overhead  of ActiveRecord.

The most useful, to those beyond myself, is probably the extending of the Ruby’s String and Fixnum classes to include ActiveRecord’s ‘ordinalize’ function. For those not familiar, it will take a String or Fixnum and return a string of the number in the following format:

1 => 1st, 2 => 2nd, 3 => 3rd, 10 => 10th, etc…

Example usage:

puts 5.ordinalize
=> 5th

For easy copy/paste, you can grab a gist of my code from my Github

The saga of my moving from OS X Snow Leopard to OS X Lion continued today as I was recompiling my myriad of MITM software packages for use on Lion. Almost everything, including my own custom software, went smoothly except Ettercap 0.7.3. I have documented my fix below for my future reference and to hopefully save someone else the four hours it took me to get everything rolling.

step 1: configure file fix.

Upon initial ./configure the following halts everything:

checking for library containing pthread_create... none required
checking whether gcc accepts -pthread... no
configure: WARNING: ***************************
configure: WARNING: * PTHREAD ARE REQUIRED !! *
configure: WARNING: ***************************


This was unexpected to say the least as my other apps didn’t seem to have any problems including pthreads. After a couple hours of wasting my life trying to figure out the problem I finally tracked it down to the Ettercap configure file itself. Simply swap out MACOSX for DARWIN on the following line (line # 28246):

ORIGINAL:

elif test "$OS" != "MACOSX" -a "$OS" != "WINDOWS"; then

FIXED

elif test "$OS" != "DARWIN" -a "$OS" != "WINDOWS"; then


step 2: libnet inclusion.

Earlier in the day I had to install libnet for one of my custom libraries, but you will also need it for Ettercap or you will get something similar to:

checking for libnet... no
configure: error: libnet >= 1.1.2.1 not found

To solve this, install libnet from http://sourceforge.net/projects/libnet-dev/ and the next time you run Ettercap’s configure file pass it the following option:

-with-libnet=/usr/local/

step 3: Seriously, another error?!? Bloody hell.

n file included from wdg.c:23:
./wdg.h:189: error: expected specifier-qualifier-list before ‘u_char’
./wdg.h:304: error: expected ‘)’ before ‘pair’
./wdg.h:305: error: expected declaration specifiers or ‘...’ before ‘u_char’
./wdg.h:312: error: expected ‘)’ before ‘pair’
wdg.c:81: error: expected ‘)’ before ‘pair’
wdg.c:82: error: expected declaration specifiers or ‘...’ before ‘u_char’
wdg.c:83: error: expected ‘)’ before ‘pair’
…

Okay, the problem here is that type ‘u_char’ is not defined in the standard C spec. On most systems there is a sys/types.h header located at /usr/include/sys/types.h. To instruct gcc to include it, pass the following to configure:

CFLAGS="-include /usr/include/sys/types.h"

step 4: A libiconv mess…

To avoid the following error during make:

Undefined symbols for architecture x86_64:
"_libiconv_open", referenced from:
_set_utf8_encoding in etterlog-ec_format.o
_utf8_format in etterlog-ec_format.o
"_libiconv_close", referenced from:
_set_utf8_encoding in etterlog-ec_format.o
_utf8_format in etterlog-ec_format.o
"_libiconv", referenced from:
_utf8_format in etterlog-ec_format.o
ld: symbol(s) not found for architecture x86_64

Install a fresh version of the iconv library to /usr/local/. Note that we are installing the 1.11 version not the latest 1.13.1 version because the 1.11 version is compatible with the iconv that ships with OS X Lion (as of the writing of this blog post on 08/12/2011). The quick and easy:

curl -O http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.11.tar.gz
tar -zxvf libiconv-1.11.tar.gz cd libiconv-1.11
./configure
make
sudo make install

Additionally, we will need to tell gcc to link to our fresh version at /usr/lib/local/ and not the /usr/lib version that ships with OS X via:

CFLAGS="-lc /usr/local/lib/libiconv.2.dylib"

step 5: No ettercap GUI for me, please… I’ll admit it, I’m an old hacker.

I feel comfortable and in control when in terminal windows. Simply put, every GUI except the iPhone’s iOS I find overwhelming and confusing. I’m sure those who began using computers post-internet feel differently, but I’m a creation of the 80′s who has been living in prompts since the age of 4. I believe in GUI’s only if they actually make you more productive for the task, such as OS X’s Expose which is absolutely brilliant for swapping between terminal windows I say that entire rant to give reason to why I did not figure out how to install the gtk+ libraries for the ettercap gui. To avoid the following:

checking for pkg-config... /usr/local/bin/pkg-config
checking for GTK_CFLAGS...
checking for GTK_LIBS...
configure: error: Package requirements
(gtk+-2.0 >= 2.0.0 pango >= 1.0
atk >= 1.0) were not met. Consider
adjusting the PKG_CONFIG_PATH environment
variable if you installed software in a
non-standard prefix.

Instruct configure to make for text mode only by using the following option:

-disable-gtk

6: libpcre (optional)  

If you have installed the pcre library and want to expand the functionality of ettercap, you will need to manually specify it’s location via configure option:

-with-libpcre=/usr/local/include

7: Finally, putting it all together.

You now should be able to compile without errors:

make clean
./configure LDFLAGS="$LDFLAGS -pthreads" \
-with-libnet=/usr/local/ -disable-gtk \
CFLAGS="-include /usr/include/sys/types.h -lc /usr/local/lib/libiconv.2.dylib" \
-with-libpcre=/usr/local/include
make
make check
sudo make install


8: Config reminder

Lastly, don’t forget that for SSL MITM on OS X you’ll need to run the following bash script beforehand using sudo:

#!/bin/sh
# redir_command_on:
if [ -a "/tmp/osx_ipfw_rules" ]; then
ipfw -q add `head -n 1 osx_ipfw_rules` fwd 127.0.0.1,$1 tcp from any to any $2 in via $3
else
ipfw add fwd 127.0.0.1,$1 tcp from any to any $2 in via $3 | cut -d " " -f 1 >> /tmp/osx_ipfw_rules
fi

and the following after you’re done using Ettercap:

#!/bin/sh
# redir_command_off
if [ -a "/tmp/osx_ipfw_rules" ]; then
ipfw -q delete `head -n 1 /tmp/osx_ipfw_rules`
rm -f /tmp/osx_ipfw_rules
fi

Both of these scripts and more important details can be found in the Ethercap config file at:

/usr/local/etc/etter.conf

Hopefully this can save someone else from going through the 28000+ lines of the configuration file, and don’t forget it’s always bad karma to use tools against anyone who helped you install them. Happy Janus attacking

—————–
now playing: Antonio Pinto – Lord of War