Problem Definition

Data validation and integrity assurance are complex subjects that have strong opinions tied to the way they are implemented. Some strongly feel that the database can handle validation using the column’s attributes (size, data type, etc) and TRIGGERS/STORED PROCEDURES can provide means to enforce complex business rules. This is true, but there are many caveats and arguments against this approach. Should a data source be performing logic? According to Oracle’s website:

“Although triggers are useful for customizing a database, use them only when necessary. Excessive use of triggers can result in complex interdependencies, which can be difficult to maintain in a large application“

Those in opposition to this approach feel that it is not the job of the database to perform logic, only to store data. So, who is right?
How should data validation be handled? Triggers or code? Honestly, both is usually the practice. The current implementations of Declarative Data Integrity Rules in the database marketplace give us simple rules to validate against (again: size, data type, nullable, etc) and if we don’t validate data in our code prior to insertion the database errors out. Though mindless and tedious, it is a straightforward task to code validation prior to executing SQL in your programming language.

I think that both opinions are correct, but both approaches are inadequate. Logic should be separated from data. However, rules about data stored in your database are really metadata. Why should programmers have to tweak code every time the table changes? Why should DBAs have to worry that programmers validation is insufficient and not have any control over the data integrity? I believe the database should dictate to us, in specific terms, what it is expecting. In addition, programmers and DBAs should be able to work together to formulate the data validation and that is should be in only one place.

Current Practice

Most of the time, a DBA will create a table with data type, length, unique and referential constraints. These rules are enforced by the database, programmers are handed back an error message which has to be parsed into a friendly error. In some shops, STORED PROCEDURES and TRIGGERS are used to validate data prior to insertion. As we pointed out above, this is not an encouraged practice and makes a data source a logic engine.

Let’s say we have a table defined as such:

id INT(8) NOT NULL,
name VARCHAR(40),
zip_code VARCHAR(10)

Programmers take the constraints defined above (i.e. “INT”, “NOT NULL”, 40, etc) and write up code validation for each of those columns. If DBAs alter the number of columns or the metadata of any one of the columns, the code needs to be updated accordingly. Conversely, if the programmers don’t properly validate the raw data in code, we could easily end up with “NOTAZIP” in our zip_code field. In this way, the technologies are seemingly blind of each other except for a tin-can phone that can pass simple messages back and forth. This is not a knock against the technology, they were made to be platform independent (in terms of interfacing) and not necessarily to know everything about each other. This is the price we have to pay for an exceedingly friendly means of storing and retrieving data.

Proposed Solution

Utilizing dynamic functions, JSON and metadata stored in column comments of my database I was able to validate data against any table with only one validation class (that isn’t very long!). I have yet to see a column or table comment be anything slightly useful (if anything at all). Up until two days ago, every time I saw the field to insert a comment in phpmyadmin I would think “I guess that’s a nice feature…”, and then move on. Then I realized that using (in MySQL) SHOW FULL COLUMN FROM {TABLE}, I can get that comment’s value back. And using my programming language (PHP), I could do the following:

$data = json_decode($comment);

I now am holding an object created from metadata to do what I will with.

The initial data I inserted into a column COMMENT was a minified representation of this:

'{	"insert_helpers": {
		"functions": {
			"func1":{
				"name":"strtotime",
				"params":{
					"param1":"+20 years"
				}
			},
			"func2":{
				"name":"str_replace",
				"params":{
					"param1":"!!",
					"param2":"!",
					"param3":"@this",
				}
			}
		}
	},
	"validators": {
		"maxlength":"10",
		"minlength":"2",
		"patterns":{
			"pattern1":{
				"pattern":"[^0-9]",
				"example":"This is data without numbers"
			}
		}
	}
}'

This was then validated with code very similar to this:

/**
*
* NOTE: I have not tested this code
* I typed directly it into notepad++ for your reading,
* not for executing, if it works, great!
*
*/
$json 				= json_decode($comment);
$helper_functions 	        = $json->{'insert_helpers'};
$validators			= $json->{'validators'};
$errors				= array();

//$row_value = $result['field'];
foreach($helper_functions as $function)
{
	$params = array();
	foreach($function->{'params'} as $param)
	{
		//insert row value
		if($param == '@this')
		{
			$param = str_replace("@this", $row_value, $param);
		}
		array_push($params, $param);
	}
	if(is_callable($function->{'name'})
	{
		$row_value = call_user_func_array($function->{'name'}, $params);
	}
}

if(is_object($validators))
{
	if(is_object($validators->{'minlength'})
	{
		if(sizeof($row_value) < $validators->{'minlength'})
		{
			array_push($errors, "$row_value is not long enough");
		}
	}
	if(is_object($validators->{'maxlength'})
	{
		if(sizeof($row_value) > $validators->{'maxlength'})
		{
			array_push($errors, "$row_value is too long");
		}
	}
	if(is_object($validators->{'patterns'})
	{
		foreach($validators->{'patterns'} as $pattern)
		{
			if(preg_match("/$pattern->{'pattern'}/", $row_value))
			{
				array_push($errors, "$row_value is not in correct format [example: $pattern->{'example'}");
			}
		}
	}
}

If you didn’t pick it up by reading through the code, these functions couldn’t care less what table you are dealing with and the same set of code works against any and all columns. In addition, the validation rules (JSON) can be handed to the client side code to perform validation prior to reaching the server. All major databases have a comment column and most modern programming languages allow behavior like the example I posted.

I will post my php class that facilitates this process soon.

Anxious to hear people’s thoughts.

Sign Up

Apply for a Google Maps API Key (free) here using your domain without a subdomain or www (this will allow you to use key with or without www and on subdomains): http://code.google.com/apis/maps/signup.html

Configuration

Put the API key in your configuration file (application.ini, config.ini, etc.) like this:

[google]
google_map_key =  YOUR_KEY

Bootstrapping

In your index.php file, add the following code:

//update CONFIG_PATH to point to your configuration file
defined(CONFIG_PATH)
or define('CONFIG_PATH', APPLICATION_PATH . '/configs/application.ini');
Zend_Registry::set('google',    new Zend_Config_Ini(CONFIG_PATH, 'google'));

This will add your api key to the registry so that it can be accessed throughout your application.

Model

In your models directory, create a new class called ‘Geocoder.php’. This will be the file that contains the application logic for retreiving coordinates on a given location.

models/Geocoder.php

/**
* Geocoder
*
* @author Robert McVey
*/
class Geocoder {
	protected $key = "";
	public function __construct($apiKey)
	{
		$this->key = $apiKey;
	}
	private function _getGeocodedLatitudeAndLongitude($address)
	{
		$client = new Zend_Http_Client();
		$client->setUri($this->getGeocodingUri());
		$client->setParameterGet('q', urlencode($address))
			->setParameterGet('output', 'json')
			->setParameterGet('sensor', 'false')
			->setParameterGet('key', $this->key);
		$result = $client->request('GET');
		$response = Zend_Json_Decoder::decode(
			$result->getBody(),
			Zend_Json::TYPE_OBJECT
		);
		return $response;
	}
	public function getCoordinates($address)
	{
		$response = $this->_getGeocodedLatitudeAndLongitude($address);
		$return = array();
		if(isset($result->Placemark[0]->Point->coordinates[1])){
			$return(
			'lat' => $result->Placemark[0]->Point->coordinates[1];
			'lon' => $result->Placemark[0]->Point->coordinates[0];
			);
		}else{
			$return = null;
		}
		return $return;
	}
	private function getGeocodingUri()
	{
		return 'http://maps.google.com/maps/geo';
	}
}
?>

Usage

Alright, that was pretty easy. Now we just have to implement this in our controller. Here is a sample model that uses the Geocoder class to return geotagged User objects.
models/User.php

class User extends Zend_Db_Table_Abstract{
	protected $_name = 'users';
	protected $key;
	protected $geocoder;

	public function init()
	{
		$this->key = Zend_Registry::get('google')->google_map_key;
		$this->geocoder = new Geocoder($this->key);

	}
	public function getUsersAndGeocode()
	{
		$result = $this->fetchAll();

		$users    = $result->_toArray();
		foreach($users as $user)
		{
			$address = "{$user['address']} {$user['city']} {$user['state']} {$user['zip']}";

			$latlon = $this->geocoder->getCoordinates($address);
			if($latlon)
			{
				$user['lat'] = $latlon['lat'];

				$user['lon'] = $latlon['lon'];
			}
		}
		return $users;
	}

}
?>

The controller part is a little too easy. We will setup a reference to our user table and create a map action. Within the map action we get all of users and geocode them. We pass that collection to our view. This file should be located here:
controllers/UserController.php

class UserController extends Zend_Controller_Action{
	protected $user_table;
	public function init()
	{
		//database connectivity stuff is up to you

		$this->user_table = new User();
	}
	public function mapAction()
	{

		$users = $this->user_table->getUsersAndGeocode();
		$this->view->users = $users;
	}
}
?>

And finally, the view. Here we will create our map div, load the Google Maps API (don’t forget to sub out your key in the script src). We are using a custom marker here (personIcon); you can add your own image here by changing the path to your image. I recommend the image be pretty small with a transparent background. Given our current setup, this file would be located at this path:
views/scripts/user/map.phtml

<div id="map">

<noscript>
<center>
<strong>To view the map feature, you need to have Javascript enabled. If you are unsure about how to turn Javascript support on, please</p>
<p>read Google's instructions found here: <a href="https://www.google.com/support/adsense/bin/answer.py?hl=en&amp;answer=12654" target="_blank">LINK</a></strong>

</center></p>
<p></noscript>
</div>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>

<!--
<script type="text/javascript" src="http://maps.google.com/maps?file=api&amp;amp;v=2&amp;amp;sensor=false&amp;amp;key=your_key"></script>
-->

<script type="text/javascript">
 //Sorry, I love jquery! If you don't use window.onload = function(){initialize();}
    $(document).ready(function(){
        initialize();

    });
var map;
var personIcon;
function initialize() {
    if(GBrowserIsCompatible){

        map = new google.maps.Map2(document.getElementById("thurtene_map"));
        map.setUIToDefault();
        map.setCenter(new google.maps.LatLng('48.858001709', '2.29460000992'), 4);
        <? endif; ?>

        personIcon            = new GIcon();
  /**
  * Change this marker to be a small transparent custom image
  *
  */

        personIcon.image      = '/img/common/custom_marker.png';
        personIcon.shadow     = "http://www.google.com/mapfiles/shadow50.png";
        personIcon.iconSize   = new GSize(32,32);
        personIcon.shadowSize = new GSize(37,34);

        personIcon.iconAnchor = new GPoint(9,34);
        personIcon.infoWindowAnchor = new GPoint(9,2);
    }
    addOverlays(map);
}

function personMarker(point, index, dto){
 //set our marker to be the custom icon we created
    markerOptions = {icon:personIcon};
    var marker = new GMarker(point, markerOptions);

/**
 * This is the content in the info bubble
 */
    GEvent.addListener(marker, 'click', function(){
       marker.openInfoWindowHtml(
            '<h3><a href="/profile/view/'+dto.user_id+'">'+dto.user_first+' '+ dto.user_last+'</a></h3>
'
            +dto.user_address+'

'
            +dto.user_city+'
'
            +dto.user_state+' '+dto.user_zip
        );

    });
    return marker;
}
/*
* Here is where we loop through our users and create new markers for each

*/
function addOverlays(map){
    <?php
  $geo = $this->users->_toArray();
        $size = sizeof($geo);

 ?>
    <? for($i = 0; $i < $size; $i++): ?>
                var latLon = new GLatLng(<?php echo $geo[$i]['lat'];?>, <?php echo $geo[$i]['lon'];?>);

                map.addOverlay(personMarker(latLon, <?=$i?>, <?=Zend_Json::encode($geo[$i])?>));
    <?php endfor; ?>
}
</script>

Post a comment if you have any questions.

CSS

.norm{
color:#fff;
background-color:#000;
}

.alt{
color:#000;
background-color:#fff;
}

PHP Code:

<?php

$style = 0;//switch to know which row we are on
echo '<table>';
while($row = mysql_fetch_array($query))
{
$name = $row['name'];//for each corresponding column name you wish to show
$add   =  $row['address'];
$email = $row['email'];

if($style == 0)
{
echo '<tr class="norm">';
echo '<td>'.$name.'</td>';
echo '<td>'.$add.'</td>';
echo '<td>'.$email.'</td>';
echo '</tr>';
$style++;
}else{
echo '<tr class="alt">';
echo '<td>'.$name.'</td>';
echo '<td>'.$add.'</td>';
echo '<td>'.$email.'</td>';
echo '</tr>';
$style--;
}
}
echo '</table>';

?>

That’s all, not much to it really.

Edit: It should be stated that this could be accomplished in a much more simple manner using the ternary operator to conditionally set the class, here is what that would look like:

<?php

$style = 'norm';//switch to know which row we are on
echo '<table>';
while($row = mysql_fetch_array($query))
{
$name = $row['name'];//for each corresponding column name you wish to show
$add   =  $row['address'];
$email = $row['email'];
echo '<tr class="'.$style.'"> ';
echo '<td>'.$name.'</td>';
echo '<td>'.$add.'</td>';
echo '<td>'.$email.'</td>';
echo '</tr>';
$style = ($style == 'norm') ? 'alt' : 'norm';
}
echo '</table>';

?>

Here is one of my htaccess files for reference

<IfModule mod_expires.c>
 ExpiresActive On
 ExpiresByType text/html "access plus 1 seconds"
 ExpiresByType image/gif "access plus 2 years"
 ExpiresByType image/jpeg "access plus 2 years"
 ExpiresByType image/png "access plus 2 years"
 ExpiresByType text/css "access plus 2 years"
 ExpiresByType text/javascript "access plus 30 minutes"
 ExpiresByType application/x-javascript "access plus 30 minutes"
</IfModule>

<ifModule mod_gzip.c>
 mod_gzip_on Yes
 mod_gzip_dechunk Yes
 mod_gzip_item_include file \.(html?|txt|css|js|php|pl)$
 mod_gzip_item_include file clientside/css
 mod_gzip_item_include file clientside/javascript
 mod_gzip_item_include handler ^cgi-script$
 mod_gzip_item_include mime ^text/.*
 mod_gzip_item_include mime ^application/x-javascript.*
 mod_gzip_item_exclude mime ^image/.*
 mod_gzip_item_exclude rspheader ^Content-Encoding:.*gzip.*
</ifModule>

<IfModule mod_deflate.so>
 SetOutputFilter DEFLATE
 AddOutputFilterByType DEFLATE text/html text/css text/plain application/x-javascript text/javascript
 BrowserMatchSetOutputFilter DEFLATE ^Mozilla/4 gzip-only-text/html
 BrowserMatch ^Mozilla/4\.0[678] no-gzip
 BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
</IfModule>

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} -s [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^.*$ - [NC,L]
RewriteRule ^.*$ /index.php [NC,L]