Most of our development is done on Vista desktops (although I also run Ubuntu of course) and we wanted a web interface for easy browsing of code so I needed to use a pervasive and well supported VCS. After much hunting around and looking at Bazaar, Mercurial and git I decided to go with the latter due to its support in the community.  There are still some reservations I have against git, which work well for Linux kernal development but not our day to day web development – the major one being that subversion would allow you to commit and update on a file by file basis where as git versions the entire repository at once as far as I am aware.

Anyway now I have an Ubuntu server that I am using for storing the repositories and as a pretty frontend I have installed InDefero which also allows us to manage tickets.  InDefero is much like the code.google.com forge to manage and look at it so it is very simple.  I like Trac but I was not happy with the existing integrations with git.

It was not all plain sailing with the InDefero install as I had to make some very minor changes in the git plugin’s core code (the git plugin comes as part of InDefero and does not need to be installed seperately).  Basically the plugin was creating new repositories with very strict permissions which meant that the neither my backup user nor the apache process user could access them.  What this meant from the frontend was that the repository statistics (disk usage) were not displayed because it could not stat the directories and my back up process could not open them to make backups.

This problem is caused by the use of permissions setting in a PHP mkdir() command in the src/IDF/Plugin/SyncGit/Serve.php on lines 94 and 193.  You will want to increase the file permission octal just enough for your other users to gain access.  It is also worth mentioning that on line 111 there is a umask() command which you might also wish to tweak to improve your access.  Also don’t forget to update all $pluf_path values in the files in the scripts directory.  The rest of the way you should be fine just following the included documentation in doc/syncgit.mdtext

So far I am enjoying using InDefero though I think I will need to hack it so that when you add a new repository/project all the users names are placed in the project members as default to save vast amounts of copying and pasting.  Our development team is not large enough to warrant only giving certain people access to certain repositories/projects.

[UPDATE] I have made a small (and nasty) hack to the source of InDefero to cause the members text area when creating a new project to be pre-filled with my member’s names.  This involved changing /src/IDF/Form/Admin/ProjectCreate.php on line 107 by setting the array element initial to the value you want to appear in the text area.

For our desktop access to git we are using msysGit, PuTTY Plink and TortoiseGit.  This combination is working well for us so far and means that even non-technical users can begin versioning their documents and push them to a central repository for backup purposes.

I use most of these commands every day to simplify my terminal interactions with an Ubuntu development box.  This is more of a personal reference but thought I would share incase you find it useful.

Sometimes you need to be able to access a remote Samba server in a secure manner from a Windows machine.  This is a relatively simple procedure on an XP SP3 machine like mine linking into an Ubuntu server pre setup with Samba file sharing.

Windows is a little bit annoying as it binds all filesharing operations to port 139 so you cannot have more than one filesharing system in operation at once.  If you were to tunnel directly across to your Samba server it would bang heads with the Windows filesharing layer.  You could just disable file sharing in Windows but that is an in elegant method and you may need access to both Windows and remote Samba shares.  So we will need to setup a new loopback adapter with a local ip address that we can tunnel Samba request through thus allowing Windows filesharing to operate normally along side Samba.  This effectively makes Windows think that it is accessing Samba shares on a seperate machine whereas a tunnel usually acts as a port on the local machine.

Add the Loopback Adapter to the Windows client machine

1. Open up the Add Hardware control panel (Start > Control Panel > Add Hardware)
2. Click next and wait for the annoying wizard to finish hunting around your system
3. Choose ‘Yes, I have already connected the hardware’
4. Then scroll to the bottom of the ‘Installed hardware’ list box and choose ‘Add new hardware device’
5. Now choose ‘Install the hardware that I manually select from a list (Advanced)’
6. Select  ‘Network adapters’
7. Under ‘Manufacturer’ you want ‘Microsoft’
8. For ‘Network Adapter’ choose ‘Microsoft Loopback Adapter’
9. You may have a to wait a little while for the adapter to be fully installed

Set the Loopback Adapters Configuration

1. Pull up the adapters properties dialogue (Start > Control Panel > Network Connections and then right click on the adapter and choose properties)
2. Disable ‘File and Printer Sharing for Microsoft Networks’
3. Highlight ‘Internet Protocol (TCP/IP)’ and click the ‘Properties’ button
   1. Choose ‘Use the following IP address’
      1. Enter ‘10.0.0.1′ for ‘IP address’
      2. Enter ‘255.255.255.0′ for ‘Subnet mask’
   2. Click the ‘Advanced’ button and on the ‘WINS’ tab
      1. Enable ‘Enable LMHOSTS Lookup’
      2. Check ‘Disable NetBIOS over TCP/IP’
4. You will now need to restart you computer even though Windows does not prompt for this step

Configure the SSH Tunnel

1. On your PuTTY session configuration dialogue choose Connection > SSH > Tunnels
2. Check/enable ‘Local ports accept connections from other hosts’
3. In ‘Source port’ enter ‘10.0.0.1:139′
4. In ‘Destination’ enter ‘localhost:139′ (127.0.0.1:139 did not work for me)

Test and Map the Connection

In the Run command console (Start > Run) enter ‘\10.0.0.1′ and you should be presented with file explorer window containing the contents of your Samba share.

So if that worked we are ready to roll, but you can give your Samba share ’server’ a more friendly name by opening ‘C:WINDOWSsystem32driversetchosts’ in your favourite editor (Vim in my case).  Scroll to the bottom and enter the following ‘10.0.0.1    samba’.  You can now access ‘//samba’ in the same way we did above via the Run dialogue.  If you have assigned the loopback device to a different subnet then you will need use the lmhosts file in the same directory instead – please see Microsoft KB Article Q105997.

Now you can Map the Samba share like any other by using the ‘Tools’ menu in a Windows file explorer window.  In the ‘Folder’ input enter ‘\samba’ or for a home directory called simon ‘\sambasimon’ (you must have enabled home directory sharing in your Samba smb.conf (/etc/samba/smb.conf)).

Memcached and APC are two tools that you can install on your server and gain almost instant gratification!  APC basically caches executions that you send to a PHP process so that the next time you ask the parser to run your script it only has to look for some pre-chewed opcode in memory rather than parsing your PHP from the disk.  APC also has another feature up its sleeve, memory object caching, which allows you to store objects such as results from a database table in memory.  Memcached works in much the same way, but it can be used as a session handler as well as a persistent store for object data.  Another advantage memcached has over APC is that it can be distributed so you could have a number of servers all maintaining memcached daemons to distribute the load.

APC is faster as a straight PHP call on the memory cache, but you can only access the memory cache from the local PHP process.  I have not tested it but I am unsure if that means you can access it from CLI PHP as well as via PHP running on Apache.  Often I will have a PHP frontend with any server scripts (for example cronjobs) written in Python so it is very handy to be able to access the cache from a central location, which with memcached is possible but not with APC as it is PHP specific.  For more information on performance and advice on when various caching methods may be more useful Peter over at MySQLPerformanceBlog has written an interest article Cache Performance Comparison.

So for code that I know will only ever be used by the local PHP process I store the object in the APC memory object store, but if I need portability or the server is under high load and I want to separate out my object cache server from my script server then I use memcached.

I have written two articles detailing the steps involved in installing both the binaries on RedHat Enterprise Linux and Ubuntu or Debian.  Currently the live servers I use are either running CentOS or RedHat and the local development server is very nice setup based upon Ubuntu.

The MySQL documentation contains a very nice set of examples and documentation on using memcached to reduce hits on the database, which includes examples in a few languages including Python and PHP.  There is also some very handy hints to be gleened from An Introduction to memcached by Jeremy Ashcraft a.k.a MrSpooky over at Search-This.

Installing APC on Debian or Ubuntu is as simple as:

user@server:/directory/$ sudo apt-get install php-apc

Now let us reboot the Apache process to enable our new cache:

user@server:/directory/$ sudo /etc/init.d/apache2 restart

APC should now be ready to run on your server.  Try running the following command to verify it is setup; you should get something in response like mine:

user@server:/directory/$ php -r ‘phpinfo();’ | grep ‘apc’
apc
MMAP File Mask => /tmp/apc.s5jA6w
apc.cache_by_default => On => On
apc.coredump_unmap => Off => Off
apc.enable_cli => On => On
apc.enabled => On => On

<…SNIP…>

Now lets move onto installing Memcached, which again is very simple:

user@server:/directory/$ sudo apt-get install memcached
user@server:/directory/$ /etc/init.d/memcached start

The PHP Memcached module can be installed through Apt-Get as well:

user@server:/directory/$ sudo apt-get install php5-memcache

Now to configure PHP to use Memcached to store the session information we need to edit our /etc/php5/apache2/php.ini file and find the lines like the following:

session.save_handler = files
;session.save_path =

and change them so that they now look like this:

session.save_handler = memcache
session.save_path = “tcp://localhost:11211?persistent=1&weight=1&timeout=1&retry_interval=15″

That just leaves us to restart the Apache2 process:

user@server:/directory/$ /etc/init.d/apache2 restart

You are now up and running with Memcached PHP sessions and APC served PHP.

As you may be aware I have recently been playing with the excellent Agavi framework and it introduced me to the interesting phing tool.  Phing can be used to automate tasks with build files that are close to interoperable with Apache Ant, which uses XML files to configure builds.  The advantage phing has for us PHP users is that it is entirely written in PHP so extending it is as simple as adding a new class.

On a few recent projects I did just that when I need the ability to perform SVN Import and access Unfuddle’s API to create a new repository on the fly.  Essentially my build file was creating a new website using a CMS and then creates a new repository for it and imports the newly built CMS automatically.  This was all written to use Unfuddle’s SVN but the add repository task should be able to be used to create a new git repository as well.  The Unfuddle side of things was completed with refactoring and building upon David Winterbottom’s work, which was originally intended to send Unfuddle messages.

Please find the code hosted on github in my phing repo.

Installing APC on Redhat is as simple as:

[user@server directory]# yum install php-pecl-apc

APC should now be ready to run on your server.  Try running the following command to verify it is setup; you should get something in response like mine:

[user@server directory]# php -r ‘phpinfo();’ | grep ‘apc’
apc
MMAP File Mask => /tmp/apc.s5jA6w
apc.cache_by_default => On => On
apc.coredump_unmap => Off => Off
apc.enable_cli => On => On
apc.enabled => On => On

<…SNIP…>

Now lets move onto installing Memcached, which again is very simple:

[user@server directory]# yum install memcached
[user@server directory]# /etc/init.d/memcached start

The PHP Memcached module can be installed through YUM as well:

[user@server directory]# yum install php-pecl-memcache

Now to configure PHP to use Memcached to store the session information we need to edit our /etc/php.d/memcache.ini file and jump to the bottom of the file where we uncomment the following lines by removing the preceeding semicolon (‘;’):

; Options to use the memcache session handler

; Use memcache as a session handler
;session.save_handler=memcache
 ; Defines a comma separated of server urls to use for session storage
;session.save_path=”tcp://localhost:11211?persistent=1&weight=1&timeout=1&retry_interval=15″

So that it now looks like this:

; Options to use the memcache session handler

; Use memcache as a session handler
session.save_handler=memcache
 ; Defines a comma separated of server urls to use for session storage
session.save_path=”tcp://localhost:11211?persistent=1&weight=1&timeout=1&retry_interval=15″

That just leaves us to restart the Apache/HTTPd process:

[user@server directory]# /etc/init.d/httpd restart

You are now up and running with Memcached PHP sessions and APC served PHP.

As you have found this page I am sure you have accidentally hit the control+s short cut whilst inside a PuTTY shell and following that no keystrokes appear to affect the session. Basically hitting ctrl+s causes PuTTY to stop executing the stream coming in from the keyboard. It does however still listen to your keystrokes and it basically adds them to a queue.

Hitting control+q will re-open the stream execution, but it is worthwhile noting that it will also execute all the queued up commands as well!

So far I have tried out Ubuntu, eeeBuntu, EasyPeasy Linux and CrunchBang Linux (all of which are Debian based) on the Asus Eee PC.  Ubuntu was a bit of an annoyance to get setup and it was troublesome trying to get all the buttons or the sound and microphone to work so I then tried EasyPeasy.  It was easy to install (it already contains the Array.org kernal) and I liked the NBR interface with its easy to use tabbed system not to mention that after using Ubuntu it was nice to be able to see most of the system dialogues on screen (Ubuntu’s dialogues were so long they fell off screen! – Hint: hold down alt and click to move dialogues without titlebars).  The major problem I had with it was that it took way too long to boot and came preloaded with a whole host of applications I will never need.

So after some research I saw a lot of good things being said about eeeBuntu, which again in the version I chose uses the NBR interface.  Also whilst it was very nice to use not everything worked, which was very annoying.  The microphone and some of the shortcut buttons did not work, which is useless if you need to use Skype.  The selection of preloaded applications was also not to my tastes (although better than EasyPeasy) and boot time was also quite painful although not as bad as EasyPeasy.  Speaking of boot times make sure you remove any SD cards from the onboard slot before booting the machine as this will add significantly to the load time of the OS.

After using eeeBuntu for some time I finally grew tired of the lag and decided it was time to do some more research into the available packages for easy installing on the Eee PC.  This is when I came across CrunchBang (#!) Linux, which is a very light weight version of Ubuntu running the Openbox shell.  This distro boots quickly, comes with a nice set of applications that I can easily build upon.  I am really pleased with the OS so far and I am even writing this post from within it.

Installation is very simple just use UNetbootin to transfer the Cruncheee image to a USB key and boot from the USB key to try out the ‘live disc’ version of the OS to see if it is for you.  If you like it then right click on the desktop and choose install from the resultant menu.

Not everything will work right of the box though.  The microphone does not record immediately and requires some tweeking to the Volume Control interface.  You can bring up the interface by pressing the shortcut Super + v (in my case the Super key is emblazoned with the Windows logo).  Now click the preferences button at the bottom of the dialogue and check the boxes beside Front Mic Boost, Capture and Input Source.  Now in the playback tab unmute Front Mic and raise the volume, on the recording tab unmute the microphone icon and raise the volume and finally on the options tab choose Front Mic from the drop down.

Unfortunately there is one further annoyance.  The volume buttons do not bring up the on screen display but Super + v will bring up the mixer which is fine for me at the moment.

Sun VirtualBox Logo

Often Linux just does it better!  Often I find myself developing a Windows machine without access to a Linux development server, but I still need to access to some of the Linux binaries and features such as cron jobs, the at command and binaries such as imagemagick, pdftotext, etc.  Some things can be emulated with ported binaries or through Cygwin, but I feel a lot more comfortable developing on a platform that is representative of the live server the web site will run on.

I have not removed XAMPP as it is very useful for developing small scripts or sites without the overhead of running a virtual machine.  Therein lies the problem with this solution – it is virtual machine based and that will mean your local development machine will suffer when the virtual machine steals computing time from the CPU or more memory.  So whilst I can use XAMPP on my netbook I am not so sure a VM will run smoothly.

Setting up a new development environment

1. Download VirtualBox 2.1 from http://www.virtualbox.org
2. Download the latest Ubuntu from http://www.ubuntu.com
   Your choice of server or desktop. I chose desktop to give me another web browser testing environment.
3. Install VirtualBox and open it up
   1. Click create new VM (Virtual Machine)
   2. Give your VM name (I called mine Development)
   3. From the drop downs choose
      1. Linux
      2. Ubuntu (only choose Ubuntu 64 bit if you are running 64 bit Windows)
   4. Choose how much of your machines physical memory the virtual machine is allowed to steal. I chose 512MB as I am running the desktop version of Ubuntu.
   5. You will need to create a new virtual hard disk – a new hard disk wizard will open
      1. Choose dynamically expanding storage
      2. Choose
         1. Give the virtual hard disk a name (mine is called Development just like my virtual machine)
         2. The location of the storage on your hard drive
4. Now your new virtual machine should be setup so right click on it and choose Start
   1. The Virtual Machine will pop up a message asking you to go through its first run wizard
   2. Choose your installation location
      1. CD/DVD ROM
      2. Image file – point this to the location of your Ubuntu installation ISO you downloaded and saved earlier
   3. This will then mount your CD image and boot from it
5. Install Ubuntu by going through its installation wizard which is very easy and does not require detail here (if you have chosen the desktop version this will be even easier because you can use the mouse!)
6. Once you have the virtual machine setup you will want the nice Linux drivers so you can have a larger screen size and for networking etc. For this you need to install the Virtual Guest Additions.
   1. Mount a new CD drive/ISO Image in your virtual machine whilst it is running from the top menu item called Devices
   2. Add the new image which you will find in your VirtualBox program folder
7. Back to your virtual machine and open up a terminal window
   1. In your terminal navigate to the CD Rom drive
   2. Execute the following command sudo sh ./VBoxLinuxAdditions-x86.run
   3. Once complete you need to reboot the machine so execute sudo reboot
8. Once the machine reboots get back into the terminal and run the following commands
   1. sudo apt-get update
   2. sudo apt-get upgrade
   3. sudo apt-get install apache2 mysql-server-5.0 php5 php5-xdebug
   4. Just to be on the safe side restart apache with sudo /etc/init.d/apache2 restart
   5. Try accessing your web server by opening a web browser in the virtual machine and typing in localhost if you chose the desktop and you should see a message like It Works!
9. We need to be able to see folders on our host machine inside the development machine quickly and easy so from the top menu bar choose Devices -> Shared Folders
   1. Add a new folder
   2. Tell it which folder to look in
   3. Give it a memorable name
   4. Tick make permanent
10. Jump back into the virtual machine and execute the following commands to add your shared folder to the virtual machine, in a terminal
    1. sudo mkdir /mnt/yourFolderNameHere (mine is /mnt/htdocs/)
    2. sudo mount -t vboxsf memorableFolderName /mnt/yourFolderNameHere
       1. Replace memorableFolderName with name you set in step 9.III
       2. Replace /mnt/yourFolderNameHere with the folder you made in step 10.I
    3. If you navigate to /mnt/yourFolderNameHere and execute ls you should see a list of the files on your host systems shared folder
    4. If that worked then we want to add our folder to the fstab file so that the mount point is loaded every time our virtual machine boots up
       1. sudo vim /etc/fstab
       2. Add the following line to the bottom of the file:
          memorableFolderName /mnt/yourFolderNameHere vboxsf defaults 0 0
          1. Replace memorableFolderName with name you set in step 9.III
          2. Replace /mnt/yourFolderNameHere with the folder you made in step 10.I
11. Now to make this development machine available over the network shut the virtual machine down by going to top menu bar Machine -> Close and choosing Power Off
    1. Right click on the virtual machine and choose settings
    2. From the left hand menu choose Network
       1. Tick Enable Adapter
       2. Adapter Type: Intel PRO/1000 T Server
       3. Attached to: Host Interface
       4. Tick Cable Connected
       5. Choose the interface on your host machine that gives you access to the outside world. You can find out which one this is by looking in your control panel.
    3. Save the settings and start your virtual machine back up again
    4. To find out the IP address of your virtual machine open a terminal execute ifconfig
       
12. You should now be able to visit your web server from your host machine by entering that IP address into your web browser.
    

Configuring our web server

1. We need to enable mod_rewrite which involves making a simple symbolic link
   1. Navigate to /etc/apache2/mods-enabled
   2. Execute sudo ln -s /etc/apache2/mods-available/rewrite.load rewrite.load
2. Reboot the server sudo /etc/init.d/apache2 restart and test it is function ok.

Setting up Mass Virtual hosts

The idea behind this step is to minimise the time it takes to setup a new host on the server. Basically all you have to do is create a new directory an away you go. (Don’t forget it will still need to appear in your hosts file).

1. We need to enable mod_vhost_alias
   1. Navigate to /etc/apache2/mods-enabled
   2. Execute sudo ln -s /etc/apache2/mods-available/vhost_alias.load vhost_alias.load
2. Now to setup the rules it will create virtual hosts by
   1. Execute sudo vim /etc/apache2/apache2.conf
   2. Go to the base of the file and add the following rules:

      http://gist.github.com/294736

      VirtualDocumentRoot sets the directory that the VHosts public directory is contained in. This will basically convert http://subdomain.localhost/ to /mnt/htdocs/subdomain and pull the relevant files through.

3. Now restart the apache process sudo /etc/init.d/apache2 restart

A side note when using this method that you should be aware of. This will affect your rewrite rules if they are placed into a .htaccess file. To avoid any problems always declare the RewriteBase rule in your .htaccess.

For example:

http://gist.github.com/294737

VirtualBox Shared Folders Permissions

Nathan brought a glaring omission from my post to the fore – thanks!  Currently you maybe having permissions issues with your shared folders because they might be being mounted as root:root.  To get them to load with a specified user and group you will need to edit your /etc/fstab file again and change ‘defaults‘ to be ‘uid=username,gid=groupname‘ – an example would be ‘uid=simon,gid=www-data‘.

A full line example would be from:
htdocs /mnt/htdocs vboxsf defaults 0 0
to
htdocs /mnt/htdocs vboxsf uid=simon,gid=www-data 0 0