Set PYTHONPATH

To set the enviornment variable PYTHONPATH in bash:
export PYTHONPATH=/path/to/modules

just setting PYTHONPATH=/path/to/modules won't work - you have to use export. If you want a variable passed on to a child processes it has to show up when you type 'env'

But since you used export - next time you start python, '/path/to/modules' will automatically be appended to the front of your sys.path


Obviously '/path/to/modules' should be the full path to whereever you're keeping your modules - something like /home/clayg/lib. (relative path's will work, but not ~/ , probably best to avoid both) You can separate multiple directories with a colon:

export PYTHONPATH=/path/to/modules:/path/to/other/modules

Setting python path is handy if, for example, you are using setuptools or distutils to install python modules on a system which you do not have root privileges.

Just download the source dist and find the directory with setup.py, then run:
python setup.py build

Then re-locate the folder with the __init__.py (usually ./build/lib/packagename) to your /path/to/modules folder.

if you want your PYTHONPATH to stick around - you should add it to .bashrc

import packagename should be good to go!

get dates from excel with python xlrd

a1_as_datetime = datetime.datetime(*xlrd.xldate_as_tuple(a1, 0))

UPDATE: Please read the discussion of the second argument to xldate_as_tuple - "datemode" in the comments section of this post before using this example. It is LIKELY that hard-coding the "datemode" option will not meet your long term needs.

I had to piece this line together from two other articles, sorry don't remember which.


Full Example:

>>> import datetime, xlrd
>>> book = xlrd.open_workbook("myfile.xls")
>>> sh = book.sheet_by_index(0)
>>> a1 = sh.cell_value(rowx=0, colx=0)
>>> print "Cell A1 is ", a1
Cell A1 is 39811.0
>>> a1_as_datetime = datetime.datetime(*xlrd.xldate_as_tuple(a1, book.datemode))
>>> print 'datetime: %s' % a1_as_datetime
datetime: 2008-12-29 00:00:00
>>>

This might make more sense if you're familiar with xlrd - A Python module for extracting data from MS Excel ™ spreadsheet files.

If you are familiar with xlrd, then the only part really worth discussing is xldate_as_tuple, which will convert the float that excel is using to store the date as something more useful, like a tuple:
(2008, 12, 29, 0, 0, 0)

Note that the first argument to the xldate_as_tuple function is the variable I defined as a1. xldate_as_tuple will not accept a cell reference 'a1' or some such thing - you have to give it the float!

The datetime module has a constructor for dates that requires at minimum three positional arguments:

datetime(

year, month, day[, hour[, minute[, second[, microsecond[, tzinfo]]]]])

This would also work:
a1_as_date = datetime.date(*xlrd.xldate_as_tuple(a1, 0)[:3])

You can pass the items of the tuple as positional arguments by prefacing the tuple with an asterisk (wtg python!)

Simple example of Threads in Python

The first time it was immediately obvious to me that there would be a significant gain from 'threading' a program I had written - was in the context of screen scraping. I had a handful of HTTP GET requests from almost 20 pages that were being processed one... after... the... other. I realized of course that if I would just start the next request before waiting on the last one to finish - the entire process would be over much more quickly.

In this example the screen scraping 'worker' function is replaced with a simpler 'random wait' function:

#! /usr/bin/env python

import sys
import threading
import time
import random

# The worker function does the processing
def worker(arg):
 arg = random.randint(2,10)
 time.sleep(arg)
 return arg

# The myThreadObj wraps the worker function in a thread
class myThreadObj(threading.Thread):
 def __init__(self, arg):
  threading.Thread.__init__(self)
  self.arg = arg
  self.value = 0
 def run(self):
  self.value = worker(self.arg)
  print 'Thread %d Ended.' % self.arg

# my array of arguments to be processed by the worker function
myArgs = range(5)

# create a myThreadObj to process each argument
myThreadList = []
for i in myArgs:
 myThreadList.append(myThreadObj(i))
 # and start it immediately
 myThreadList[i].start()

# wait for all threads to finish
for each in myThreadList:
 each.join()

print 'All threads have completed.'

for i in myArgs:
 print "myThreadList[%d] = %d" % (i, myThreadList[i].value)

The myThreadObj wrapper should accept whatever arguments you normally pass to the worker, and when the worker is completed - it will store the returned value in 'self.value'

The .join() function blocks until the .isAlive() method would return false. I process each thread iteratively to verify that all have completed. It doesn't matter if .join() blocks for 8 seconds while it's waiting on the first thread, or if it gets to a thread that's already been completed for 6 seconds cause an earlier .join was waiting on a previous thread that took longer. The point is that, by the time all of the .join() statements complete - ALL THREADS HAVE FINISHED.

Once the threads are done we expect myThreadObj.value to contain the return value of the worker function.

If your 'worker' function is something like an API call, or database query - anything with some built in lag from a system that's designed to serve multiple simultaneous requests - as long as you can queue them up - threading will provide a significant improvement.

e.g.

clayg@m-net:~$ cat nonthread.py
#! /usr/bin/env python

import sys
import threading
import time
import random

# The worker function does the processing
def worker(arg):
arg = random.randint(2,10)
time.sleep(arg)
return arg

myArgs = range(5)

for i in myArgs:
print "myThreadList[%d] = %d" % (i, worker(i))
clayg@m-net:~$ time ./nonthread.py ; echo ; time ./simplethread.py
myThreadList[0] = 5
myThreadList[1] = 5
myThreadList[2] = 10
myThreadList[3] = 2
myThreadList[4] = 9

real    0m31.073s
user    0m0.045s
sys     0m0.024s

Thread 4 Ended.
Thread 0 Ended.
Thread 1 Ended.
Thread 2 Ended.
Thread 3 Ended.
All threads have completed.
myThreadList[0] = 5
myThreadList[1] = 10
myThreadList[2] = 10
myThreadList[3] = 10
myThreadList[4] = 2

real    0m10.078s
user    0m0.045s
sys     0m0.030s

Parsing a list of numbers in Python

I find that I often need to get a selection of numbers in a range as input. I'm using Python more and more these days it seems, so I needed to port this classic function over. I must have done this 4 weeks ago - I'd been meaning to put it up here.

The valid input will be a comma separated list of integers, which could possibly contain a 'range' defined as "x-y" - where x and y are both integers.

I tried not to make any special stipulation for the order of these integers, or even that the input string would not contain bad characters.

Here it is:

#! /usr/local/bin/python
import sys
import os

# return a set of selected values when a string in the form:
# 1-4,6
# would return:
# 1,2,3,4,6
# as expected...

def parseIntSet(nputstr=""):
  selection = set()
  invalid = set()
  # tokens are comma seperated values
  tokens = [x.strip() for x in nputstr.split(',')]
  for i in tokens:
     try:
        # typically tokens are plain old integers
        selection.add(int(i))
     except:
        # if not, then it might be a range
        try:
           token = [int(k.strip()) for k in i.split('-')]
           if len(token) > 1:
              token.sort()
              # we have items seperated by a dash
              # try to build a valid range
              first = token[0]
              last = token[len(token)-1]
              for x in range(first, last+1):
                 selection.add(x)
        except:
           # not an int and not a range...
           invalid.add(i)
  # Report invalid tokens before returning valid selection
  print "Invalid set: " + str(invalid)
  return selection
# end parseIntSet

print 'Generate a list of selected items!'
nputstr = raw_input('Enter a list of items: ')

selection = parseIntSet(nputstr)
print 'Your selection is: '
print str(selection)

When trying to copy this from someone else I came across a similar function written in Ruby, in case you needed that instead.

olpc default no password

I received an OLPC in the mail today. It's on loan from a family member who got it as a gift for her daughter. I was told "It's not a REAL computer! - can you fix it?"

Yes, send it to me.

Whatever it takes right? I couldn't wait to tear into this thing.

SSH is up and running by default out the gate. Just go to the "Neighborhood View" and connect to your in home wireless AP and you can hit ssh right away:

[olpc@xo-14-7F-A0 ~]$ telnet 192.168.1.118 22
Trying 192.168.1.118...
Connected to 192.168.1.118.
Escape character is '^]'.
SSH-2.0-OpenSSH_4.5

Which is cool, but there's a catch - you can't login.

The default user is named 'olpc' - this user, as well as root DO NOT HAVE A DEFAULT PASSWORD.

The password file is shadow'd:
[olpc@xo-14-7F-A0 ~]$ egrep '^(root|olpc)' /etc/passwd
root:x:0:0:root:/root:/bin/bash
olpc:x:500:500::/home/olpc:/bin/bash

But if you look at the hash in /etc/shadow you see olpc's "current password hash" is "!!" - which is not valid:
[olpc@xo-14-7F-A0 ~]$ egrep '^(root|olpc)' /etc/shadow
root::13880:0:99999:7:::
olpc:!!:14148:0:99999:7:::

root's password is just plain empty!

Which is handy on the local system:
[olpc@xo-14-7F-A0 ~]$ whoami
olpc
[olpc@xo-14-7F-A0 ~]$ su
bash-3.2# whoami
root

But kind of awkward when you try to login via ssh:
login as: olpc
olpc@192.168.1.118's password:
Access denied

The tricky part about that invalid "!!" hash is that it's neither a blank password nor a valid password. The olpc user can't even change it's OWN password because there is NO CURRENT VALID PASSWORD:
[olpc@xo-14-7F-A0 ~]$ passwd
Changing password for user olpc.
Changing password for olpc.
(current) UNIX password:
passwd: Authentication token manipulation error

So we decide move on to root.

Root has an 'empty' password. Which is valid - but it had me raise en eyebrow as well.

login as: root
root@192.168.1.118's password:
Access denied


I was slightly forgiving after my investigation into the default configuration of OpenSSH.
bash-3.2# egrep '#Permit(Root|Empty)' /etc/ssh/sshd_config
#PermitRootLogin yes
#PermitEmptyPasswords no

Because these configuration lines are commented out, the ssh server goes with the standard OpenSSH settings. Which will NEVER allow the super user to to authenticate directly over ssh, and furthermore would not allow ANY user with an empty password to authenticate anyway.

To login to your OLPC XO Laptop via ssh you will need to:
1) open a local terminal
2) su to root
3) set a valid password for the 'olpc' user
4) connect to the laptop as the 'olpc' user via ssh using the new password you just created

e.g.

[olpc@xo-14-7F-A0 ~]$ whoami
olpc
[olpc@xo-14-7F-A0 ~]$ su
bash-3.2# passwd olpc
Changing password for user olpc.
New UNIX password:
BAD PASSWORD: it is too short
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
bash-3.2# exit
exit
[olpc@xo-14-7F-A0 ~]$ ssh olpc@192.168.1.118
The authenticity of host '192.168.1.118 (192.168.1.118)' can't be established.
RSA key fingerprint is d4:5a:b0:30:0d:eb:61:9c:df:c9:32:0e:ab:ff:51:9e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.118' (RSA) to the list of known hosts.
olpc@192.168.1.118's password:
[olpc@xo-14-7F-A0 ~]$

Stay tuned - More to come!

Display free memory in Linux / Ubuntu

Short Answer:
free -m | awk 'NR==3 {print $4 " MB"}'

The command 'free' displays some formatted information from /proc/meminfo.
The throw '-m' displays these numbers in rounded megabytes.


# free -m
         total      used     free   shared     buffers     cached
Mem:      4049      3982       67        0          16       3530
-/+ buffers/cache:   435     3614
Swap:     6142        53     6088

It's strange, cause I always figured the "first row, third column" - right under 'FREE' - would be the amount of free memory on a Linux system. But that number was always low after I'd been running awhile.


And it turns out that it is NOT really what I was looking for. A more accurate representation of the memory being used by your applications and available for new processes is displayed in the SECOND line.

In addition to the memory that is actually being USED by the kernel and processes resident in memory - Linux also reserves memory to allocate to processes as 'buffers' AND uses pretty much any left over memory to hold "cached" files.

Looking only at the top line...
total = all memory in the system (4GB on this server)
used = all memory currently in use/reserved by running processes and the OS
free = total - used
shared = memory being shared by multiple processes (deprecated?)
buffers = memory reserved by the OS to alloc as buffers when process need them (aka the 'heap')
cached = recently used files being stored in ram (THANK YOU LINUX!)

Here's a simple example I found to show off the power of 'caching':

for i in 1 2 ; do free -o; time grep -r foo /usr/bin >/dev/null 2>/dev/null; done

So really the buffers would be allocated to a running process if it asked for them anyway, and the memory being used to cache copies of recently used files would be released immediately if it makes sense to allocate the RAM elsewhere. So all that memory is 'available'.

Using these definitions:

When thinking about 'how much memory is really being used' - I want to calculate:
'used' - ('buffers' + 'cached')

When thinking about 'how much memory is really free' - I want to calculate:
'free' + ('buffers' + 'cached')

With this in mind, the meaning of the second row header form the output of the Linux command "free" (-/+ buffers/cache:) makes more sense...

Free is doing some light lifting for us, using the formula's above to display:
"minus buffers and cache" for the used column
and
"plus buffers and cache" for the free colum

So when you run free on Linux - the amount of free memory is always displayed right there in the second row, third column. Hence the 'Short Answer'...

- Much respect to Mike Griffin who got me thinking about this.

DD-WRT Xbox LIVE open NAT

When I first setup my DD-WRT router I had problems with Xbox LIVE reporting strict NAT and then moderate NAT.

To achive "open" NAT status on Xbox LIVE your best bet is ALWAYS UPnP.

Not DMZ.
Not even setting up manual LIVE port forwarding.

Both of these methods fall apart if you have two Xboxs. UPnP allows the Xbox(s) to decide the best ports to use, when they should be open and when to close them.

By default the DD-WRT v24 firmware disables UPnP.

To Enable UPnP on DD-WRT v24:
NAT / QoS -> UPnP ->
UPnP Service: Enabled

Click Apply. You may need to reboot your xbox for changes to take effect.

In order for UPnP to work as intended the client device (the xbox) must be using DHCP to obtain an IP address from the UPnP enabled router. To verify your Xbox is using DHCP, as opposed to assigning itself a static ip - open the dashboard and navigate to:
System -> Network Settings -> Edit Settings ->
Select "IP Settings"
Verify "Automatic"

To see what your Xbox thinks about your NAT try the "Live Connection Test"
System -> Network Settings -> "Test Xbox LIVE Connection"

Easy DD-WRT wireless bridge

So this is me connecting to a dd-wrt on the other side of the room - no wires


First you have to be able to connect to both routers on different ip's on the same subnet via wired connection.

primary gateway - 192.168.1.1
remote client bridge - 192.168.1.2

If you're all set up and rocking it with your primary gateway on 192.168.1.1 - but you need to add this new device which defaults to the same ip into the mix....
Just log into your current device and change it's ip to .3 - then plug in the new router, navigate to it on .1 - change it .2 - go back to the original on .3 and put it back on .1

You *may* want to go ahead and disable DHCP on the remote client bridge while your in there, but only if you're not going to immediately configure wireless.

Remote client bridge (192.168.1.2) wireless setup:

Step 1:
Wireless -> Basic Settings

Wireless Mode: Client Bridge
Wireless Network Name (SSID): "SSID of primary gateway / wireless AP"

click Apply

Step 2:
Wireless -> Wireless Security
configure wireless settings to match your primary gateway's wireless config

Apply changes

That's it, unplug your router and test the connection!

I *highly* recommend you START with 64bit/10 digit hex WEP. Go ahead and change your settings on the primary gateway / wireless AP if you have to. Once you have that working by all means upgrade to a higher security state. But, do your self a favor and start simple to make sure you have it working.

DD-WRT applyuser.cgi hangs

What to do if you're running into the issue with the DD-WRT admin page hanging on the applyuser.cgi script when you try to save changes.

SYMPTOM:
when you click 'apply' you get directed to the address -

http://192.168.1.1/applyuser.cgi

Which is followed by a blank white screen. Navigating back just proves your changes were not affected.

SOLUTION:
restart the browser - it's probably just a cookie with a hung up user session or something, but I couldn't find it. Other suggestions would be very welcome...

HOW TO: DD-WRT on Asus WL-500G Premium V2

Our goal is simple:


Put DD-WRT on an Asus WL-500G Premium V2

Our process is straight forward

1. establish ip connectivity to router
2. download dd-wrt firmware
   
3. download Asus WL-500G Premium V2 utilitypack
4. reboot router in 'restore' mode
5. disable windows firewall (really?! Yes, I'm afraid so)
6. install/run Firmware Restoration utility
7. browse to new firmware and click upload
   
8. wait for router to reboot automatically
9. re-enable windows firewall

Step 1:
My current router was already running on 192.168.1.1 - so I moved it to 192.168.1.2 left my pc on dhcp. I suppose you could always configure a static ip 192.168.1.??? and a subnet of 255.255.255.0 - The point is you have to be able to ping the asus on 192.168.1.1 !

Step 2:
DD-WRT download section:
http://www.dd-wrt.com/dd-wrtv3/dd-wrt/downloads.html

stable -> dd-wrt.v24 SP1 -> Consumer -> ASUS -> WL500g-Premium ->
dd-wrt.v24_mega_generic.bin

Step 3:
GAWD ASUS SITE SUCKS! I hate javascript sometimes...

Asus download section:
http://support.asus.com/download/download.aspx

Product: Wireless
Series: AP/Router
Model: WL-500gP V2

Direct WL-500gP V2 download section
Direct WL-500GP V2 Utility download

*I also found it on the included CD, just run D:\Utility\setup.exe

Step 4:

• Unplug the router
• Press and hold small black 'restore' button on back next to antenna
• Plug in router and continue to hold 'restore'
• Wait for the power 'icon' on the front to blink on/off once per second
• Release 'restore' button - power icon should continue to blink

Success:
You should still be able to ping the router on 192.168.1.1 while it is in this mode.

Step 5:

Disable Windows Firewall

You can skip this step for now, but you're going to have to do it eventually. The ASUS firmware Restoration Utility will run, but won't upload while the windows firewall is active.

Start -> Control Panel -> Windows Firewall -> Off

Step 6:

Extract the UT_WL500gPv2_4008.zip that you downloaded earlier and run the setup.exe to install the Asus WL-500G Premium V2 utilitypack

Run the Firmware Restoration Utility:
Start -> Program Files -> ASUS Utility -> WL-500gP V2 Wireless Router -> Firmware Restoration

Step 7:

Click Browse and navigate out to:
dd-wrt.v24_mega_generic.bin

Click Upload...

It kind of bugs me that it didn't ASK what ip I wanted to upload this firmware to, but I guess I think it 'knows'...

While the progress bar is pushing around you may see some blinking on the router at first, but after awhile it just sits there:

Should take about a minute for the progress bar to quit, and then you can click close.

Step 8:

Really, just sit there... for like 45-60 seconds. I know, seems like forever. Eventually the modem will reboot, and you'll get the 'AIR' light lit up and I noticed I could ping it again. While you waiting don't forget Step 9!

Step 9:

Don't forget to turn back on windows firewall!!!

DONE. Navigate back to http://192.168.1.1/ and setup the user:


You haven't really LIVED until you've ssh'd into your router:

login as: root
DD-WRT v24 mega (c) 2008 NewMedia-NET GmbH
Release: 07/27/08 (SVN revision: 10012:10013)
root@192.168.1.1's password:
==========================================================

____ ___ __ ______ _____ ____ _ _
| _ \| _ \ \ \ / / _ \_ _| __ _|___ \| || |
|| | || ||____\ \ /\ / /| |_) || | \ \ / / __) | || |_
||_| ||_||_____\ V V / | _ < | | \ V / / __/|__ _|
|___/|___/ \_/\_/ |_| \_\|_| \_/ |_____| |_|

DD-WRT v24
http://www.dd-wrt.com

==========================================================


BusyBox v1.11.1 (2008-07-27 19:56:11 CEST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

root@DD-WRT:~# cat /proc/cpuinfo
system type : Broadcom BCM5354 chip rev 2
processor : 0
cpu model : BCM3302 V2.9
BogoMIPS : 239.20
wait instruction : no
microsecond timers : yes
tlb_entries : 32
extra interrupt vector : no
hardware watchpoint : no
VCED exceptions : not available
VCEI exceptions : not available
root@DD-WRT:~#

Note the superuser username. Instead of the username you created in the DD-WRT initial configuration page, you have to use 'root' as the login username - but still the same password.

ASUS WL-500g Premium V2 and DD-WRT

So I picked the ASUS WL-500g Premium because...

1. I knew it was supported by DD-WRT
2. Lots of memory (32 MB)

I think I read somewhere that it was actually a pretty good piece of hardware for this kind of thing... but it doesn't matter - it looks cool:

Not going to be needing this:



ASUS WL-500g Premium V2 default IP:
192.168.1.1

ASUS WL-500g Premium V2 default Login:
Username: admin
Password: admin

So this is what we're trying to get rid of:

OK, so now how to get DD-WRT onto this thing...

HOW TO: DD-WRT wireless bridge

Got my package from newegg today:

Two ASUS WL-500g wireless routers. I plan to load them with DD-WRT and build a stable wireless bridge.

So I'd like to create a complete guide here, as I work - more updates to follow!

Update:
HOW TO: DD-WRT on Asus WL-500G Premium V2
Easy DD-WRT wireless bridge

Best Bash Script EVER - "Just In Case"

clayg@m-net:~$ cat bin/jic
#!/bin/bash
if [ $# -eq 0 ]
then
echo "usage: jic [filename]"
exit 1
fi
if ! [ -e $1 ]
then
echo "$1 does not exist!"
exit 1
fi
ext=$(date +%y%m%d.%H%M%S)
jic=$1.$ext
cp $1 $jic
chmod -x $jic
echo "SAVED: $jic"

Credit goes to a sourpuss I used to work with by the name of Chuck Carson, who gave me the idea. Where ever you are big guy - thanks.

The script creates a backup copy of a file you are about to modify:
# jic /etc/sysconfig/iptables
would create a copy called /etc/sysconfig/iptables.YYMMDD.HHMMSS

you know... just in case

Also handy for versioning revisions to scripts you're working on - never now when you're latest hack will turn out to be a total bust.

Also - go get yourself a free shell account on http://www.arbornet.org/

can linux tell me how many open memory slots i have

yes it can:
sudo lshw

lshw is a small tool to extract detailed information on the hardware configuration of the machine. It can report exact memory configuration, firmware version, mainboard configuration, CPU version and speed, cache configuration, bus speed, etc.

So if you're running linux and you need to know your graphics chip set, your motherboard manufacturer or model number... but you're too lazy to open the case and look ;)

Try the Linux utility List Hardware.

Use nmap to scan for ssh servers on subnet

Nmap ("Network Mapper") is a free and open source utility for network exploration or security auditing. If you don't have it - get it:

sudo apt-get install nmap

There's also a GUI called Zenmap that I use sometimes. But I'm trying to learn my way around the powerful command line interface...

Here's a simple example that will scan all computers on your 255.255.255.0 subnet and report any devices listening on port 22 - the default for SSH. All of this along with the version of SSH that the server is running is output to a text file 'sshservers':

nmap -p 22 --open -sV 10.0.0.0/24 > sshservers

They say "necessity is the mother of invention" - See what happened was the DHCP server at worked assigned one of my machines a new ip and I wanted to try and track it down remotely. The only thing I could really think of that I knew - was that it was running sshd and it was somewhere in on this one 'VLAN'. To narrow down possible candidates I wanted to gather a list of all the machines on this one subnet listening on port 22. Nmap is perfect for this kind of job!

Lets break it down

nmap : the executable name
-p 22 : specifies the port to test
--open : suppress output for clients that are not listening
-sV : display the version string reported by the scanned server
10.0.0.0/24 : the target network, could have been 192.168.0.0/24
(/24 specifies a subnet of 255.255.255.0, look up slash notation)
> sshservers : redirects standard output to a file named 'sshservers'

Here was me:

Interesting ports on [ipaddressremoved]:
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 4.7p1 Debian 8ubuntu1.2 (protocol 2.0)
MAC Address: [macaddressremoved] (Intel)
Service Info: OS: Linux

Rambling

I bought a domain - it's clayg.info

Godaddy said it was only $0.99 - but it ended up being $1.19 after some kind of ICANN tax. I also got a dyndns.org account that I'm going to use keep a pointer to my computer at home. And I'm going to have a few cname records for static private ips around the office - 10.0.0.0 stuff. I'd never even thought of it, but if you're on a network where you don't control the dns records - it IS surprisingly handy.

I bought a used e machine that I'm going to clean up into a project server. I want to install a few different things. FreeBSD, Apache, PHP, JBoss, MySQL & PostgreSQL. I'll have it setup here at home and use the godaddy/dyndns address like my own little web host.

I got that Ubuntu and Windows XP secure synergy tunneled through ssh setup working. But I didn't take very good notes. The highlights were definitely:
a) use CopSSH for the ssh server on the windows box
b) start the synergy client from /etc/gdm/PreSession/Default

Just after the fi, and before the exit
ssh -f -N -L 24800:[synergy server ip]:24800 [username]@[synergy server ip]
synergyc -f localhost > /dev/null &

Obviously setting up ssh keys for passwordless authentication is key. Generate the key on the client, and then append the public key to the ~/.ssh/authorized_key file on the server.

The SWT project is coming along. I've updated the engine to better support multiple 'views' in the right pane. I'll post the source here when I get a chance.

Oh and I'm a dad:

Deploy simple swt application in eclipse

Once you have your program running in Eclipse, at some point your going to want to share it. It is easy to do - despite all the tutorials about Java Web Start and jnlp .xml files - you don't need a web server. You can build an old fashioned platform specific .jar file very easily.

This tutorial is for SWT v3.3 (and above? I only test on Windows & Linux)

Under your workspace/myprojectname folder create a folder called "build"
under that make a folder for each OS you want to distribute, for example:
/home/clayg/workspace/LeftTabs/build/LeftTabs-linux
/home/clayg/workspace/LeftTabs/build/LeftTabs-winxp

Get your hands on the latest SWT release for the OS you're building for:
Linux - swt-3.3.1.1-gtk-linux-x86.zip
Windows - swt-3.3.1.1-win32-win32-x86.zip

Once you extract that archive you'll see a file called "swt.jar" right in the root - that's your platform specific implementation of all the SWT classes your neat little app is using. It has to go in the working directory of the .jar you're going to create for your application.

So copy the swt.jar from the linux release into:
/home/clayg/workspace/myprojectname/build/myprojectname-linux
and copy the swt.jar from the windows release into:
/home/clayg/workspace/myprojectname/build/myprojectname-winxp

(C:\Documents and Settings\clayg\workspace\myprojectname\build\myprojectname-[platform])

Then go into eclipse. Open your project, we're going to create a manifest for your application's java archive. The manifest tells the java runtime which class to execute and what classes it depends on (I'm pretty sure that's basically what it's doing?).

Right click on your project in the Package Explorer and select New -> File. Name it myprojectname-manifest.txt (LeftTabs-manifest.txt). Fill it with the following goodness:

Manifest-Version: 1.0
Class-Path: swt.jar
Main-Class: myprojectname
[blank line]

Replace myprojectname with the name of the class that contains your main method, which is hopefully also the name of the project, and also more than likely your one and only source file? Replace [blank line] with you guessed - a blank line. I tried it without the blank line, and you really do need it - LAME.

Now right click on your project again, and this time selected Export. In the wizard, under Java - pick "JAR File" and click Next. Your project should already be selected, under "Select the export destination:" click Browse next to "JAR File"

You want to create a file called myprojectname-[platform].jar under the folder:
/home/clayg/workspace/myprojectname/build/myprojectname-[platform]
so the first time I did this for the linux platform, I ended up with
/home/clayg/workspace/LeftTabs/build/LeftTabs-linux/LeftTabs-linux.jar
the second time, when I did this again for windows, I got:
/home/clayg/workspace/LeftTabs/build/LeftTabs-winxp/LeftTabs-winxp.jar

Click Next, make sure "Export class files with warnings" is selected, click Next again.

Now choose "Use existing manifest from workspace" and click Browse. Select the myprojectname-manifest.txt, click OK. Now just hit Finish to build your application.

You need the whole myprojectname-[platform] directory to run your app.
In windows:
double click the myprojectname-winxp.jar file
In linux open a console in the myprojectname-linux folder, and run:
$java -jar myprojectname-linux.jar
from the command prompt

In the process of building my LeftTabs application for windows, I discovered a bug that didn't show up when I was building it under Linux. I believe the end result is actually a more elegant solution.

Line 374:
text.addListener(SWT.FocusOut, new Listener()
{ ...

became:
text.addListener(SWT.Deactivate, new Listener()
{ ...

The purpose of this Listener was to trigger the Modify Event for the view Widget which would tell the main application that the data in the view should be written back to the selected item in the tree. I was doing this when Text item in the view LOST FOCUS. But apparently in windows this happens after the new item is selected in the Tree - which caused the data in the tree to loose sync with the data in the view.

Anyway, vivir es aprender, I didn't know an event SWT.Deactivate existed. Works great.

SWT Java - Left Hand Tabs - with Drag and Drop custom treeItem.setData()

I'm a month behind my planned development because I had to build the foundation of my application before I could even start.

Here's hoping someone else gets a head start.

LeftTabs.java is a simple expandable application based on the SWT Java framework that has a "list" of "items" on the left, and on the right a "view" that will display their data.


The list is implemented by a tree. You can assign your own custom data type to each treeItem. You can create new items with a right-click pop-up menu. You can delete or drag and drop any item in the tree. Your custom data associated with the treeItem will be meticulously preserved. You can update the data in the view on the right and the changes will be synced back to your custom data type in the selected treeItem - automatically.

It's not all that surprising that more templates like this aren't available. It was rather difficult to but together. But there was lots of resources that went into making this possible. Many elements in this code are based on examples taken from: www.java2s.com


/*
* Basic template for app with left hand tabs
* and custom pane on right
* by clay.dot.gerrard.at.gmail.dot.com
*/
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;

import org.eclipse.swt.SWT;
import org.eclipse.swt.widgets.Composite;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Event;
import org.eclipse.swt.widgets.Listener;
import org.eclipse.swt.widgets.Menu;
import org.eclipse.swt.widgets.MenuItem;
import org.eclipse.swt.widgets.Shell;
import org.eclipse.swt.widgets.Text;
import org.eclipse.swt.widgets.Tree;
import org.eclipse.swt.widgets.TreeItem;
import org.eclipse.swt.graphics.Point;
import org.eclipse.swt.graphics.Rectangle;
import org.eclipse.swt.widgets.MessageBox;
import org.eclipse.swt.dnd.*;
import org.eclipse.swt.layout.*;

public class LeftTabs {

// left hand "tab" tree
static Tree tree;
// right hand viewer
static CompositePane view;
// temporary TreeItem used to hold data during Drag & Drop
static TreeItem sourceTreeItem;
// keep track of number of treeItems created
static int count = 1;

public static void main(String[] args) {
// setup the display & shell & layout
final Display display = new Display();
final Shell shell = new Shell(display);
FillLayout fillLayout = new FillLayout();
// columns
fillLayout.type = SWT.HORIZONTAL;
// break 'em up a little
fillLayout.spacing = 3;
fillLayout.marginHeight = 3;
shell.setLayout(fillLayout);

// And here's the tree
tree = new Tree(shell, SWT.BORDER);
// filled with some data
for (int i = 1; i <= 3; i++) {
TreeItemData myData = new TreeItemData();
myData.Name = "Item" + i;
myData.field1 = myData.Name + " Data";
TreeItem item = new TreeItem(tree, SWT.NONE);
count++;
item.setText(myData.Name);
item.setData(myData);
}

// And here's the right hand pane
view = new CompositePane(shell);

// right click menu on the tree
tree.addListener(SWT.MenuDetect, new Listener() {
public void handleEvent(Event event) {
Menu menu = new Menu(shell, SWT.POP_UP);
// NEW ITEM
MenuItem item_new = new MenuItem(menu, SWT.PUSH);
item_new.setText("New Item");
// when click "New Item", add an item to tree
item_new.addListener(SWT.Selection, new Listener() {
public void handleEvent(Event e) {
TreeItem[] selection = tree.getSelection();
int index;
if (selection.length != 0)
index = tree.indexOf(selection[0]) + 1;
else
index = 0;
TreeItem item = new TreeItem(tree, SWT.NONE, index);
TreeItemData myData = new TreeItemData();
myData.Name = "Item" + count++;
myData.field1 = myData.Name + " Data";
item.setText(myData.Name);
item.setData(myData);
view.setData((TreeItemData) item.getData());
tree.setSelection(item);
}
}); // end item_new event

// DELETE ITEM
MenuItem item_delete = new MenuItem(menu, SWT.PUSH);
item_delete.setText("Delete Item");
// when click "Delete Item", delete selected item
item_delete.addListener(SWT.Selection, new Listener() {
public void handleEvent(Event e) {
TreeItem[] selection = tree.getSelection();
int index = tree.indexOf(selection[0]);
for (int i=0; i < selection.length; i++){
selection[i].dispose();
}
// item deleted need new selection
TreeItem[] items = tree.getItems();
if (items.length != 0) {
if (index > items.length-1) {
// it would be better to select an item closer to the deleted item?
view.setData((TreeItemData) items[items.length-1].getData());
tree.setSelection(items[items.length-1]);
} else {
view.setData((TreeItemData) items[index].getData());
tree.setSelection(items[index]);
}
}
}
}); // end item_delete

menu.setLocation(event.x, event.y);
menu.setVisible(true);
while (!menu.isDisposed() && menu.isVisible()) {
if (!display.readAndDispatch())
display.sleep();
}
menu.dispose();
}
}); // end right click menu

// when a new treeItem is selected, update the view
tree.addListener(SWT.Selection, new Listener() {
public void handleEvent(Event event)
{
view.setData((TreeItemData) event.item.getData());
}
}); // end update view with tree item data

// when the tree looses focus make sure things are in order
tree.addListener(SWT.FocusOut, new Listener() {
public void handleEvent(Event event) {
TreeItem[] selection = tree.getSelection();
// no item selected - view data is ambigous, possible data loss
if (selection.length == 0) {
MessageBox messageBox = new MessageBox(shell, SWT.ICON_ERROR);
messageBox.setText("Possible Loss of Data.");
messageBox.setMessage("Think about what you did to cause the view data and the tree selection to get out of sync, and decide how you want your program to behave!");
messageBox.open();
//*/
TreeItem[] items = tree.getItems();
if (items.length != 0) {
view.setData((TreeItemData) items[0].getData());
tree.setSelection(items[0]);
} else {
// no items in view?! you can't delete the last item...
TreeItemData myData = new TreeItemData();
myData.Name = "Item" + count++;
myData.field1 = myData.Name + " Data";
TreeItem item = new TreeItem(tree, SWT.NONE);
item.setText(myData.Name);
item.setData(myData);
view.setData((TreeItemData) item.getData());
tree.setSelection(item);
}
//*/
}
}
}); // end tree loose focus

// setup DragSource
DragSource source = new DragSource(tree, DND.DROP_COPY);
source.setTransfer(new Transfer[] { TreeItemDataTransfer.getInstance() });

source.addDragListener(new DragSourceAdapter() {
public void dragStart(DragSourceEvent event) {
TreeItem[] selection = tree.getSelection();
if (selection.length > 0 && selection[0].getData() != null) {
event.doit = true;
sourceTreeItem = selection[0];
} else {
event.doit = false;
}
} // end dargStart

public void dragSetData(DragSourceEvent event) {
if (TreeItemDataTransfer.getInstance().isSupportedType(event.dataType))
event.data = sourceTreeItem.getData();
} // end dargSetData

public void dragFinished(DragSourceEvent event) {
if (event.doit) {
sourceTreeItem.dispose();
}
sourceTreeItem = null;
} // end dragFinished
}); // end DragSource

// setup DropTarget
DropTarget target = new DropTarget(tree, DND.DROP_COPY);
target.setTransfer(new Transfer[] {TreeItemDataTransfer.getInstance() });

target.addDropListener(new DropTargetAdapter() {
public void dragEnter(DropTargetEvent event) {
event.detail = DND.DROP_COPY;
} // end dragEnter

public void dragOver(DropTargetEvent event) {
event.feedback = DND.FEEDBACK_SCROLL;
// if the drop target is a specific item in the tree
if (event.item != null) {
TreeItem item = (TreeItem) event.item;
Point pt = display.map(null, tree, event.x, event.y);
Rectangle bounds = item.getBounds();
// give visual cue of drop location to user
if (pt.y < bounds.y + bounds.height/2) {
event.feedback |= DND.FEEDBACK_INSERT_BEFORE;
} else {
event.feedback |= DND.FEEDBACK_INSERT_AFTER;
}
} else {
// set event.item to last item in list & set feedback to after
}
} // end dragOver

public void drop(DropTargetEvent event) {
try {
if (event.data == null) {
event.detail = DND.DROP_NONE;
return;
}

TreeItem newItem;
// if the dropTarget is a specific item in the tree
if (event.item != null) {
TreeItem selection = (TreeItem) event.item;
Point pt = display.map(null, tree, event.x, event.y);
Rectangle bounds = selection.getBounds();
//TreeItem[] items = tree.getItems();
// find index of selection
int index = tree.indexOf(selection);
// insert newItem at index
if (pt.y < bounds.y + bounds.height/2) {
// insert before
newItem = new TreeItem(tree, SWT.NONE, index);
} else {
// insert after
newItem = new TreeItem(tree, SWT.NONE, index+1);
}
} else {
// no specific item selected, drop at the end
newItem = new TreeItem(tree, SWT.NONE);
}

TreeItemData myType = (TreeItemData) event.data;
newItem.setText(myType.Name);
newItem.setData(myType);
// set selection otherwise view data gets out of sync
tree.setSelection(newItem);
} catch (RuntimeException e) {
e.printStackTrace();
}
} // end drop
}); // end DropTarget

// write view data back to tree - see CompositePane.Update()
tree.addListener(SWT.Arm, new Listener() {
public void handleEvent(Event event) {
tree.getSelection()[0].setData((TreeItemData) view.getData());
}
});

// wrap it up
shell.open();
while (!shell.isDisposed()) {
if (!display.readAndDispatch())
display.sleep();
}
display.dispose();
}

} // class LeftTabs

class TreeItemData {
// all the data to be held by each treeItem
String Name;
String field1;
} // new members must be explicitly handled in nativeToJava & javaToNative

// Transfer class for handling DND of TreeItemData
class TreeItemDataTransfer extends ByteArrayTransfer {
private static final String TreeItemData_TRANSFER_NAME = "TreeItemData_TRANSFER";
private static final int TreeItemData_TRANSFER_ID = registerType (TreeItemData_TRANSFER_NAME);
private static TreeItemDataTransfer instance = new TreeItemDataTransfer();

public static TreeItemDataTransfer getInstance() {
return instance;
}

protected String[] getTypeNames() {
return new String[] { TreeItemData_TRANSFER_NAME };
}

protected int[] getTypeIds() {
return new int[] {TreeItemData_TRANSFER_ID};
}

public void javaToNative (Object object, TransferData transferData) {
if (object == null || !(object instanceof TreeItemData))
return;

TreeItemData myType = (TreeItemData) object;

if (isSupportedType(transferData)) {
try {
ByteArrayOutputStream stream = new ByteArrayOutputStream();
DataOutputStream out = new DataOutputStream(stream);
// write out each member in your custom dataType
out.writeUTF(myType.Name);
out.writeUTF(myType.field1);
out.close();

super.javaToNative(stream.toByteArray(), transferData);
} catch (IOException e) {
e.printStackTrace();
}
}
} // end javaToNative

public Object nativeToJava (TransferData transferData) {
if (isSupportedType(transferData)) {
byte[] buffer = (byte[]) super.nativeToJava(transferData);
if (buffer == null)
return null;

TreeItemData myType = new TreeItemData();

try {
ByteArrayInputStream stream = new ByteArrayInputStream(buffer);
DataInputStream in = new DataInputStream(stream);
// read in each member in your custom dataType
myType.Name = in.readUTF();
myType.field1 = in.readUTF();
in.close();
} catch (IOException e) {
e.printStackTrace();
return null;
}
return myType;
} else {
return null;
}
} // end nativetoJava

} // end TreeItemDataTransfer

// Composite widget for defining the right hand "view"
class CompositePane extends Composite {

// local TreeItemData, to keep track of changes
static TreeItemData viewData;

// declarations of pane view widgets
static Text text;

public CompositePane (Composite c) {
// setup layout of the pane
super(c, SWT.NONE);
this.setLayout(new FillLayout());

// put widgets in the pane
text = new Text(this, SWT.BORDER | SWT.MULTI | SWT.WRAP);

// add a loose focus listener to EVERY *EDITABLE* widget
text.addListener(SWT.Deactivate, new Listener()
{
public void handleEvent(Event event)
{
// sync changes in view back to dataum
Update();
}
});
} // CompositePane constructor

public void setData (TreeItemData n)
{
viewData = n;
// populate the pane's elements with the viewData
text.setText(viewData.field1);
}

// update viewData, then notify listener that it should sync to the tree
public void Update() {
viewData.field1 = text.getText();
this.notifyListeners(SWT.Modify, new Event());
}

public TreeItemData getData()
{
return viewData;
}

} // class CompositePane

I'm not happy with my options for displaying code in google blogger, and more than a little disappointed in this temporary solution. Expect a blog post to come out of that. I'm also trying to work on a secure crossplatform Snergy config that's giving me trouble so it may be worth a post.

how to delete last command from bash history

If you have ever typed something into a command prompt that you wished you hadn't - you may find it useful to know that you can delete it from ~/.bash_history very easily.

The command:

history -d offset

will delete the history entry at position offset.

# history
1 cd
2 history
3 ls -alhF
4 history
5 wget username:password@private.ftp.com/secret/file.tar.gz
6 history

so to delete the wget command (which contains a password) - just use:
history -d 5

# history -d 5
# history
1 cd
2 history
3 ls -alhF
4 history
5 history
6 history -d 5
7 history



This is (hopefully) the final update to an OLD post.  Sorta "pre-StackOverflow" internet.  

Checkout this question instead:
http://superuser.com/questions/384366/remove-a-certain-line-from-bash-history-file

^posted in the comments

The below is preserved only for hysterical accuracy


But suppose you KNOW you're about to enter a command you don't want to go into history. It'd be nice if you could just tack a little "hideme" modifer onto the front or tail of your command and be done with it. Unfortunately from what I've been able to google there is no such feature built into history or bash.

Naturally I made one.

TMP=$(history | tail -1 | awk '{print $1}') && history -d $TMP && \
paste_in_shell_and_replace_this_with_whatever_you_want_to_hide
And naturally someone smarter than me came along and found a better way to do it - THANKFULLY they posted a comment here to help us out (thanks Mitch!):

history -d $((HISTCMD-1)) && \
paste_in_shell_and_replace_this_with_whatever_you_want_to_hide

Rather than holding down backspace, you may find it useful to know that in bash Ctrl-W will delete from the cursor to the beginning of the previous word.

What I don't get, is that according man bash HISTCMD should be the CURRENT history number:

HISTCMD
The history number, or index in the history list, of the current command.

and yet in ALL my tests $HISTCMD is the "index in the history list, of the current command" +1

But it can still lead to two two useful aliases:

alias hideme='history -d $((HISTCMD-1))'
alias hideprev='history -d $((HISTCMD-2)) && history -d $((HISTCMD-1))'

Dig the sneaky:

# history
1 cd
2 history
3 ls -alhF
4 history
5 history
6 history -d 5
7 history
8 vi .bashrc
9 history
# echo password && hideme
password
# echo password
password
# hideprev# history
1 cd
2 history
3 ls -alhF
4 history
5 history
6 history -d 5
7 history
8 vi .bashrc
9 history
10 history

I know the blog's kinda been on Linux kick lately - some of that is coming from the new job - I'm using Linux more. But, I've been working on a little project in Eclipse - Java/SWT - and I'm getting to a point where I may have some useful learnings to post coming out of that. Or maybe not...

Internet Tablet, and all my xbox's are running fine...

I'd like to throw out some props to Ivie for sending me an email about one of my posts that she read. I try to post stuff that I myself have trouble finding out there on the interwebz - so it's always nice to hear from someone that finds it useful - thanks Ivie!

ubuntu disable hardware

I had a second sound card that kept getting picked up by apps under ubuntu as my primary audio device. In windows I sometimes use the game port on this pci sound card, but in general I don't even care if Linux detects it.

In ubuntu gnome, under System->Preferences->Sound
I had something like this:


In an unrelated effort, I had only recently discovered an important linux command - lsmod

The command list modules will display information about all loaded kernel modules. Kernel modules are dynamically loaded kernel functions, such as a device driver.

When I looked through the list of loaded kernel modules on my system:

lsmod | less

I found this:

snd_ens1370 21536 0
gameport 16776 1 snd_ens1370
snd_ak4531_codec 9856 1 snd_ens1370
snd_pcm 80388 5 snd_ens1370,snd_intel8x0,snd_ac97_codec

There was no doubt in my mind - I needed to stop the kernel from loading the module snd_ens1370.

I had considered just deleting it all together:

$ locate 1370
/lib/modules/2.6.22-14-generic/kernel/sound/pci/snd-ens1370.ko
/usr/src/linux-headers-2.6.22-14-generic/include/config/snd/ens1370.h
/usr/share/alsa/cards/ENS1370.conf

But it turns out there is a more elegant way.

To disable hardware in ubuntu - blacklist the module.

Take a look at, /etc/modprobe.d/blacklist

To disable my Ensoniq sound card in ubuntu - I added the following lines:

# disable my PCI ensoniq sound card
blacklist snd_ens1370

then just reboot...

how does bash work

When evoked the Borne Again Shell can launch one of two different types of interactive shells, login or non-login.

These descriptions are based on my experiences with a variety of Debian & Fedora based environments - each has their own peculiarities in their implementation of bash. I'll do my best to be "generic"

When you login via tty, ssh, or su --login - your $SHELL is launched for you. This is called a "login" shell, and it will automatically source the following files:

1) /etc/profile

then

2) ~/.profile or ~/.bash_profile
(but generally not both, one will take "precedence")

When you launch /bin/bash by some other means (xterm, konsole, su, or even typing /bin/bash) it is considered non-login and it WILL source .bashrc

It is VERY common for ~/.profile (or ~/.bash_profile) to source ~/.bashrc by either:

. ~/.bashrc

or

source ~/.bashrc

You can use source either way - in both interactive and non-interactive shells.

/etc/profile may source lots of other files. Sometimes /etc/bashrc (or /etc/bash.bashrc) and sometimes even ~/.bashrc - so watch out!

Your system may not have all these files - or they may have different names. Try:

ls -al /etc/ | grep bashrc
ls -al /etc/ | grep profile
ls -al ~ | grep bashrc
ls -al ~ | grep profile

to get some different ideas.

the command "su -" is short hand for "su --login root" or "switch user to root and launch a login shell"
it is NOT "switch to super user and load environment variables" (even if that may be the case on some systems)

3RLOD Error 0020 X Clamp Replacement

And so, it was my turn:


For a used 360, I'm actually pretty happy it lasted as long as it did. Manufacture date was 2/06 - so there's nothing holding me back now.

Funny thing, this is actually how my FIRST xbox got modded. When it broke, it was out of warranty, so I had nothing keeping me from opening it up. XBMC is the best thing I ever did for my entertainment center.

In this case the interwebz says I need an X-Clamp Replacement - so you know... whatever.

I went with RBJ's MKIII method. Which is extremely well documented, so I didn't really take many pictures. (Update: Last I checked RBJ had taken down his tutorial, which is too bad - I sent him a message. Lamma has a good one up, which is essentially the same thing. However, it should be noted I did NOT buy the kit he's selling - although I'm sure it's worth every penny - you can get the parts anywhere. It's the work of removing the x-clamp, not the tools/equipment you use, that fixes your xbox.)

I used the following hardware from Lowe's:
2 packs - M5-.8x20mm - Oval Head Phillips - Sales # 138573 - $.92 each pack (Qty:2)
2 packs - M5-.8x16mm - Oval Head Phillips - Sales # 138566 - $.68 each pack (Qty:2)
5 packs - 5mm Flat Washers - Part # 138319 - $.56 each pack (Qty:10)


I spent probably more time than I should have cleaning the old thermal paste off the chips & heat sinks. Goo-Gone and 99% alcohol. Then I used a razor to spread a very thin even layer of some generic thermal paste on the CPU and GPU. I just used what I had lying around instead of the "artic silver" stuff everyone talks about.

Drilling the holes in the case was surprisingly pretty easy - I used a 1/4" bit for a pilot hole, then opened it up to 3/8's. Using the tape to hold the bolts in place while I got the board in place also went pretty smooth.

I accidentally screwed the CPU head sink down backward on my first try putting things back together (heat tube AWAY from the GPU) - but once I got everything screwed down... It booted up on the first try - green lights all around.

Been running great for 3 days - posted success.

BTW, I start the new job @ Rackspace on Monday.