Vietnam - 4. Simplifying

How do I simplify the following problems?
Super long functions
Make staffs understand that ONE function should do ONE and only ONE specific task! Normally, the a function should contain no more than 20 LOC for our PHP web projects, excluding comments and line separators. Unless, it's a controller function.

Big fat controllers
To simplify the project, we should keep the controller skin, model fat!
Why? All the logics are in the controller, we shouldn't make it more complicated than it should be!

Logics found in the views
All the logics should be in the controller. The purpose of the view is displaying the data. The decoration should be in the CSS. Sound simple, but making all staffs understand this concept is not as simple as it should be. I need to give them concrete examples.

Zend_Form not being used
Most of our staffs do not understand Zend_Form. They think it's too complicated. I have to show them a concrete example of Zend_Form. The code from controller has been reduce from about 100 LOC to about 20 LOC.
Here are some features from Zend_Form:

• Has a lot of built in validators. Can validate number, string, phone number, credit card...
• Separate the code from HTML
• Can be decorated with Zend_Decorator
• Reusable code
• Easier to debug

Zend_Form is quite complex, example and documentation can be found here:
http://akrabat.com/php/simple-zend_form-example/
http://framework.zend.com/manual/en/zend.form.html

Variable name has no meaning
This is very hard to simplify! One simple rule is no abbreviation.
As most of the data type we are dealing with are string, number, array, Zend_Db_Table_Row, Zend_Db_Table_Rowset, therefore, I defined the following rules for our developers:

• Zend_Db_Table_Row - variable must end with Row. E.g.: $userRow
• Zend_Db_Table_Rowset - variable must end with Rowset. E.g.: $userRowset
• Array - we use plural to indicate array or self-describing name. E.g.: $userRows, $usernames, $userProfile
• String - normal variable name. E.g.: $username, $firstName,...
• Number - normal variable name. E.g.: $userAge, $earning,...

The most important concept I've taught our staffs about naming convention are:

• Variable/Class names should be noun unless you have a good reason for it not to be noun.
• Function names must be verb.
• Always use self-describing names.
• Never use shortcut.

English is a second language for our developers, therefore, we have to limit the vocabularies being used.

• All functions return a boolean must start with is, has, can.
• Function start with set accepts only ONE input parameter. E.g.: setEmail($email), setAge($age), setPassword($password)...
• Function start with update accepts an array as input parameter. E.g.: updateUserProfile($userProfile), updatePostalAddress($postalAddress), updateCreditCardInfos($creditCardInfos)...
• We have a lot of problems with naming the action controller. Here are some examples of inconsistency: getUsersAction, listUsersAction, showUsersAction,... I forced all developers to name it show. In fact, any action name from controller that involve displaying data must be started with show. Similar for edit/update/change & delete/remove/del & add/insert/insert-new/new...

Magic numbers
Magic numbers can be eliminated by moving to a constant file.

Global variables
Global variables can be used but it need to be well explained & self-describing. However, we should try to minimize the use of global variables.

It cost me a lot of time to put our staffs on track. Here are some fundamental rules for my staffs:

• All functions must have comments and @author.
• Global variables must have comments.
• Write short simple functions.
• No magic numbers.
• Variable and function names must be self-described.
• No abbreviation.
• If you think that section of code you've wrote is complex, then it is! Find a way to simplify it.

Everything should be made as simple as possible, but not simpler. Albert Einstein.

Vietnam - 3. Welcome to the real world

Please check my previous post if you haven't read it Standardizing.

After a few lectures about standardizing, simplifying, planning... Our staffs start working on the new project. I am quite busy designing other system whilst taking care of the servers. I let the staff work alone for one whole week!

Week 1:
Objective: Get the job done using the standards.

Results:
I expect the front end of the site to be completed. Yes, it is! I tested the site and found a few bugs. I dug into the code to fix it. Omg, I cannot believe that I could not understand what's going on! Here are some of the problems found:

1. Big fat controllers
2. Logic found in views
3. Models not being used much
4. Zend_Form is not used at all
5. Only a handful number of functions is found
6. Variable name has no meaning
7. LOOP in LOOP in LOOP
8. Hardly see any comment
9. Global variables are widely used
10. Magic numbers everywhere

I decided to inspect the CSS code, I found the following problems:

1. Inline style everywhere
2. ID selectors are widely used instead of class selectors
3. Many ID selectors having the same name
4. Normalization not being used
5. Short hand not being use much
6. Magic numbers everywhere, they are not even nice number. Imo, nice number are number that easy to remember and can divide many other numbers. Odd number should be avoided. 

The biggest problem is none of the standards specified is followed.

Well, one thing I cannot deny is they work hard and willing to learn. Remember, professionals built the Titanic, amateurs built Noah's Ark.

Vietnam - 2. Standardizing

Existing Office
There isn't any problem with setting up the office Vietnam as we already have an existing office with about 20 CSR. The CSR are working around the clock. We moved in, share the existing facilities.

SVN & Linux
Initially, I setup the subversion and local development environment. As a Linux geek, I forced everyone to use Linux, Ubuntu distribution. I don't see the need for Windows at all. Here are some reasons I think Linux is better. At least for our PHP developers.

• Free, easy to install
  
• No drivers needed
  
• Haven't got any complaint from our staffs about virus
• Represent our server environment
• Staffs cannot do much beside working
• The list is endless...
  

There are many cases you need Windows. Please do not argue here as I don't want to end up fighting the never ending war between Windows and Linux.

Coding Standard
I divided the coding standard to 3 sections:

1. PHP - Most of our web projects are based on the Zend Framework. Hence, we use the Zend Framework coding standard.
   
2. Database - Coding standard for SQL is a little bit tricky. I couldn't find any standard that suit our needs. I have to invent my own standard. Here are some guidelines:
   
   
   
   • Database name: lowercase, usually project name
   • Table name: plural, uppercase first letter, then CamelCase. Only the last word is plural. E.g.: UserProfiles
   • Reference table name: we use the word Maps to indicate. E.g.: UserGroupMaps
     
   • Field name: singular, uppercase first letter, then CamelCase. E.g.: FirstName, UserId
     
   • Primary key: should be the singular table name follow by Id. E.g.: UserId
     
   • Avoid using `-` and spaces
   • All table, field, functions, triggers, stored procedure names should be as meaningful as possible.
     
   • Never use abbreviation. Some people understand others don't, so avoid using it.
   • SQL keywords must be CAPITALIZED
3. CSS - All our web projects are using jQuery extensively and we are planning to keep it that way. jQuery/UI using hyphen `-` as word separator for CSS, we will stick with it. The key is making the class name as meaningful as possible.

The key here is to make everyone using the same simple standard. Talking the same language. Thinking of the same concept.

I long know simplicity is the key! Can I simplify it?

Vietnam - 1. Outsourcing

After working as a CTO (I considered myself more like a System Designer) for a small telco Filanity Corp. for 6 months. I realized the importance of outsourcing.

Our head office is located in Huntington Beach, California. However, we'd like to reduce the software development cost. I recruited a team of 10 developers in Vietnam.

• 1 x HTML/CSS/Photoshop Designer
• 1 x Photoshop/Flash Designer/Developer
• 8 x PHP Developers

I've live in Australia for almost 20 years. However, my Vietnamese is pretty good. There isn't any problem communicating with the Vietnamese developers.

The cost for the developers has been reduced significantly. A senior PHP developers would only cost us about $500 USD/month.

When hiring people in Vietnam, the following issues should be considered:

• Extremely hard to get Linux programmer.
• Spaghetti code.
• "Instant noodle" code. Most programmers are trained for that purpose. Most of the projects they've done are small projects. No maintenance is required.
  
• They can get the tasks done, but other people or teammate cannot understand.
  
• Don't be surprised if you see function with more than 100 lines of complex code.
• Wheels will be reinvented, many are not even round!
  

Well, they are quite smart. They just need to be trained!

After all, practice made perfect! Actually, only perfect practice made perfect!

SVN: write post-commit hooks using PHP

Linux shell scripting might be difficult for a number of people. Hence, I am writing this to help those who are not familiar with shell scripting.

There are cases when you want to see the changes immediately after committing your work to svn server. There are 2 ways I can think of:

1. Go to svn server and do a manual svn update. I do not recommend this repetitive task.
2. Creating a post-commit hook that do the svn update automatically.  

This guide will help you create a post-commit hook using PHP script.

Requirements:

• First, you need to know how to execute PHP file from command line. 

Assumptions:

• You should have multiple repositories. Unless you have only one project. By having multiple repositories, you can have a different version control for each repository.

Go to your svn repository, in my case it's located under:

/home/svn/projectX

You should see a number of directories under projectX, let's look at hooks and forget the rest. You should see the following files:

• post-commit.tmpl
• post-lock.tmpl
• post-revprop-change.tmpl
• post-unlock.tmpl
• pre-commit.tmpl
• pre-lock.tmpl
• pre-revprop-change.tmpl
• pre-unlock.tmpl
• start-commit.tmpl

For this guide we only interested in post-commit hook. This hook will execute after some body commit their work to svn server. You can easily do the same for all other hooks.

Now, create a new file call post-commit. This file will be executed after svn commit is executed.

#!/usr/bin/php5
<?php
/* Debugging - you never know if this file is being executed! Check the log */

//file_put_contents('/tmp/svn.log',  date("H:i:s"));

/* Location of your web directory */
chdir('/home/web/svn');

/* All projects check out should be here */
$repos = scandir('.'); 

/* Only do svn update if the repository committed is found in the /home/web/svn */
if ($repo = basename($_SERVER['argv'][1]) && in_array($repo, $repos)) {
    `/usr/bin/svn update $repo --username username --password password --quiet`;
}

Make sure this file have executable permission and its belong to correct user /group. If you can't make it works, try:

chmod 0777 post-commit

This would allowed everyone to execute post-commit script.

Note: there are 2 arguments passed to the post-commit script:

1. $_SERVER['argv'][1]  (the path to this repository)
2. $_SERVER['argv'][2]  (the number of the revision just committed)
    

Transfer MySQL database to another server with minimal downtime through replication

There are cases when you need to transfer MySQL data from one server to another powerful server. Here are some of the common methods:

1. Backup/restore MySQL database
   Backup data from the source server and restore on the destination server. This is the safest way to transfer but it will consume some time depending on the amount of data you have. Usually, it's taking much longer for restoring than backing up. Especially, if we are using InnoDB database engine.
2. Replication MySQL server
   Personally, I prefer this method as the service downtime is minimal. If you don't know MySQL Replication, please read it.

Furthermore, this document will only focus on transferring MySQL data to another server using MySQL Replication.

Objectives:

• Transfer MySQL data from source server to destination server.

Requirements:

• Can perform MySQL Replication.

Assumptions:
Assume that you have bin-log enabled on the source (master) server and configurations are configured correctly on both servers. Otherwise, please read my MySQL Replication post.

Procedures:

1. Disable write on MySQL source server. Temporarily disable write queries on the source server in order for us to do a backup with consistence data. All write queries are stored on log file. Write operation will be resumed once write is enabled.
   

   mysql> FLUSH TABLES WITH READ LOCK;

2. Get the binary log position
   

   mysql> SHOW MASTER STATUS;

   If we see something like this, then we are on the right track:
   

   +------------------+----------+--------------+------------------+
   | File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
   +------------------+----------+--------------+------------------+
   | mysql-bin.000001 |      106 |              |                  |
   +------------------+----------+--------------+------------------+
   1 row in set (0.00 sec)

   Please take note of the position (106)! This is the log position where we start backing up the data.
   
3. Backup the databases on source server using mysqldump
   

   mysqldump --routines --triggers --single-transaction --quick -uroot -p --all-databases > all-dbs.sql

4. Enable write on MySQL source server. This allowed data to be written to MySQL database again.
   

   mysql> UNLOCK TABLES;

5. Transfer the database backup file to MySQL destination server.
6. Restore the database on the destination server
   

   mysql -uroot -ppassword mysql < /path/to/all-dbs.sql
   Note: -ppassword: no space between -p & password

7. Set the binary log position
   

   mysql> CHANGE MASTER TO
   MASTER_HOST='192.168.0.1',
   MASTER_USER='slave_user',
   MASTER_PASSWORD='password',
   MASTER_LOG_FILE='mysql-bin.001',
   MASTER_LOG_POS=106;

8. Start replication
   

   mysql> SLAVE START;

9. Wait until destination (slave) server catch up (synced) with source (master) server.
   
   1. Stop MySQL service on destination server:
      

      Debian based:  service mysql restart
      Red Hat based: /etc/init.d/mysqld restart

   2. All database access must now point to the destination server. For minimal downtime, we should configure port-forwarding using iptables on the source server. This mean all incoming traffic from MySQL port (3306) should be forwarded to the destination MySQL server. Setup iptables port-forwarding is beyond the scope of this document.
   3. Now we have to make sure all servers accessing the source MySQL server have to point destination MySQL server. After this is done, we can disable port-forwarding on the source server.

By now, the source server should be able to power off. This method is by far the quickest way I could think of.

MySQL Replication

Two years ago, I had to learn MySQL replication because we had to transfer our databases to a new server with minimal downtime. If it's performed correctly, the downtime could be under 60 seconds. In my case, the downtime was about 5 minutes because of my carelessness. Nonetheless, MySQL replication can be used for many purposes including:

1. Scalability - if most of your queries are read.
2. Data security
3. Analytics - there are cases when you have complex queries and you do not want to put the stress on the production server.
4. Long-distance data distribution

Whatever your reason for coming here. Let's cut it short and start configuring.

Configuring the master server

1. Master server must have binary log enabled
2. Unique server-id must be configured

Edit my.cnf, for

• Red Hat based server: /etc/my.cnf
• Debian based server: /etc/mysql/my.cnf

Insert the following lines under [mysqld]:

[mysqld]
log-bin   = mysql-bin
server-id = 1

Make sure the following lines must be commented out. Otherwise, slave servers won't be able to access the master:

#skip-networking
#bind-address    = 127.0.0.1

Now, restart MySQL service:

Debian based:  service mysql restart
Red Hat based: /etc/init.d/mysqld restart

Create replication user on master server

1. Log into MySQL server as root
   

   mysql -uroot -p

2. Now, on MySQL shell create the replication user
   

   mysql> GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY 'password';

   mysql> FLUSH PRIVILEGES;

Configuring the slave server

1. Unique server-id must be configured. Remember, none of the slave/master servers can have the same server-id. Server-id enabled communication between the master and slave servers.
2. Enabling binary log is not compulsory but it recommended for data backup and crash recovery. There might be a case when you want this slave server to be the master server for other slaves.
3. The following configurations can be set/changed whilst MySQL service is running:
   
   • master-host: master IP address or hostname.
   • master-user: MySQL user account with replication privilege.
   • master-password: replication user password.
   • master-connect-retry: seconds to retry to connect after network failure.

Edit my.cnf and insert the following lines under [mysqld]:

[mysqld]
log-bin               = mysql-bin
server-id             = 2
master-host           = 192.168.0.1
master-user           = slave_user
master-password       = password
master-connect-retry  = 60

Now, restart MySQL service.

Preparing to replicate
Most of the cases, we have our master database running long before the slave. Hence, the master database should have a lot of data. We must import data from master to slave before replicating. There are two ways:

1. LOAD DATA FROM MASTER:
   use this if you are willing to have the master server being locked during the importing operation.
   
   1. Log into MySQL server as root:
      

      mysql -uroot -p

   2. Now, importing and replicating from MySQL shell:
      

      mysql> LOAD DATA FROM MASTER;

   Done! Life cannot be simpler than this!
   
   
2. Common procedure:
   
   Tasks on master server
   
   1. Enabling READ LOCK from master server for mysqldump later. We want to make sure that data will not be changed while we dumping the databases:
      

      mysql> FLUSH TABLES WITH READ LOCK;

   2. Now, get the log sequence number:
      

      mysql> SHOW MASTER STATUS;

      If we see something like this, then we are on the right track:

      +------------------+----------+--------------+------------------+
      | File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
      +------------------+----------+--------------+------------------+
      | mysql-bin.000001 |      106 |              |                  |
      +------------------+----------+--------------+------------------+
      1 row in set (0.00 sec)

      Please take note of the position (106)! This is the log position where we start backing up the data.
   3. Backup up the database from Linux shell:
      

      mysqldump --routines --triggers --single-transaction --quick -uroot -p --all-databases > all-dbs.sql

   4. Unlock the master database so data can be written:
      

      mysql> UNLOCK TABLES;

   5. Now, transfer SQL file (all-dbs.sql) to slave server.
   
   Tasks on slave server
   
   1. Import data to slave server:
      

      mysql -uroot -ppassword mysql < /path/to/all-dbs.sql
      Note: -ppassword: no space between -p & password

   2. Stop slave from replicating:
      

      mysql> SLAVE STOP;

   3. Now, we have to manually set the log sequence position:
      

      mysql> CHANGE MASTER TO
      MASTER_HOST='192.168.0.1',
      MASTER_USER='slave_user',
      MASTER_PASSWORD='password',
      MASTER_LOG_FILE='mysql-bin.001',
      MASTER_LOG_POS=106;

   4. Finally, start replicating:
      

      mysql> SLAVE START;

By now, our slave server should be replicating from the master server.