Problem Definition

Data validation and integrity assurance are complex subjects that have strong opinions tied to the way they are implemented. Some strongly feel that the database can handle validation using the column’s attributes (size, data type, etc) and TRIGGERS/STORED PROCEDURES can provide means to enforce complex business rules. This is true, but there are many caveats and arguments against this approach. Should a data source be performing logic? According to Oracle’s website:

“Although triggers are useful for customizing a database, use them only when necessary. Excessive use of triggers can result in complex interdependencies, which can be difficult to maintain in a large application“

Those in opposition to this approach feel that it is not the job of the database to perform logic, only to store data. So, who is right?
How should data validation be handled? Triggers or code? Honestly, both is usually the practice. The current implementations of Declarative Data Integrity Rules in the database marketplace give us simple rules to validate against (again: size, data type, nullable, etc) and if we don’t validate data in our code prior to insertion the database errors out. Though mindless and tedious, it is a straightforward task to code validation prior to executing SQL in your programming language.

I think that both opinions are correct, but both approaches are inadequate. Logic should be separated from data. However, rules about data stored in your database are really metadata. Why should programmers have to tweak code every time the table changes? Why should DBAs have to worry that programmers validation is insufficient and not have any control over the data integrity? I believe the database should dictate to us, in specific terms, what it is expecting. In addition, programmers and DBAs should be able to work together to formulate the data validation and that is should be in only one place.

Current Practice

Most of the time, a DBA will create a table with data type, length, unique and referential constraints. These rules are enforced by the database, programmers are handed back an error message which has to be parsed into a friendly error. In some shops, STORED PROCEDURES and TRIGGERS are used to validate data prior to insertion. As we pointed out above, this is not an encouraged practice and makes a data source a logic engine.

Let’s say we have a table defined as such:

id INT(8) NOT NULL,
name VARCHAR(40),
zip_code VARCHAR(10)

Programmers take the constraints defined above (i.e. “INT”, “NOT NULL”, 40, etc) and write up code validation for each of those columns. If DBAs alter the number of columns or the metadata of any one of the columns, the code needs to be updated accordingly. Conversely, if the programmers don’t properly validate the raw data in code, we could easily end up with “NOTAZIP” in our zip_code field. In this way, the technologies are seemingly blind of each other except for a tin-can phone that can pass simple messages back and forth. This is not a knock against the technology, they were made to be platform independent (in terms of interfacing) and not necessarily to know everything about each other. This is the price we have to pay for an exceedingly friendly means of storing and retrieving data.

Proposed Solution

Utilizing dynamic functions, JSON and metadata stored in column comments of my database I was able to validate data against any table with only one validation class (that isn’t very long!). I have yet to see a column or table comment be anything slightly useful (if anything at all). Up until two days ago, every time I saw the field to insert a comment in phpmyadmin I would think “I guess that’s a nice feature…”, and then move on. Then I realized that using (in MySQL) SHOW FULL COLUMN FROM {TABLE}, I can get that comment’s value back. And using my programming language (PHP), I could do the following:

$data = json_decode($comment);

I now am holding an object created from metadata to do what I will with.

The initial data I inserted into a column COMMENT was a minified representation of this:

'{	"insert_helpers": {
		"functions": {
			"func1":{
				"name":"strtotime",
				"params":{
					"param1":"+20 years"
				}
			},
			"func2":{
				"name":"str_replace",
				"params":{
					"param1":"!!",
					"param2":"!",
					"param3":"@this",
				}
			}
		}
	},
	"validators": {
		"maxlength":"10",
		"minlength":"2",
		"patterns":{
			"pattern1":{
				"pattern":"[^0-9]",
				"example":"This is data without numbers"
			}
		}
	}
}'

This was then validated with code very similar to this:

/**
*
* NOTE: I have not tested this code
* I typed directly it into notepad++ for your reading,
* not for executing, if it works, great!
*
*/
$json 				= json_decode($comment);
$helper_functions 	        = $json->{'insert_helpers'};
$validators			= $json->{'validators'};
$errors				= array();

//$row_value = $result['field'];
foreach($helper_functions as $function)
{
	$params = array();
	foreach($function->{'params'} as $param)
	{
		//insert row value
		if($param == '@this')
		{
			$param = str_replace("@this", $row_value, $param);
		}
		array_push($params, $param);
	}
	if(is_callable($function->{'name'})
	{
		$row_value = call_user_func_array($function->{'name'}, $params);
	}
}

if(is_object($validators))
{
	if(is_object($validators->{'minlength'})
	{
		if(sizeof($row_value) < $validators->{'minlength'})
		{
			array_push($errors, "$row_value is not long enough");
		}
	}
	if(is_object($validators->{'maxlength'})
	{
		if(sizeof($row_value) > $validators->{'maxlength'})
		{
			array_push($errors, "$row_value is too long");
		}
	}
	if(is_object($validators->{'patterns'})
	{
		foreach($validators->{'patterns'} as $pattern)
		{
			if(preg_match("/$pattern->{'pattern'}/", $row_value))
			{
				array_push($errors, "$row_value is not in correct format [example: $pattern->{'example'}");
			}
		}
	}
}

If you didn’t pick it up by reading through the code, these functions couldn’t care less what table you are dealing with and the same set of code works against any and all columns. In addition, the validation rules (JSON) can be handed to the client side code to perform validation prior to reaching the server. All major databases have a comment column and most modern programming languages allow behavior like the example I posted.

I will post my php class that facilitates this process soon.

Anxious to hear people’s thoughts.

Sign Up

Apply for a Google Maps API Key (free) here using your domain without a subdomain or www (this will allow you to use key with or without www and on subdomains): http://code.google.com/apis/maps/signup.html

Configuration

Put the API key in your configuration file (application.ini, config.ini, etc.) like this:

[google]
google_map_key =  YOUR_KEY

Bootstrapping

In your index.php file, add the following code:

//update CONFIG_PATH to point to your configuration file
defined(CONFIG_PATH)
or define('CONFIG_PATH', APPLICATION_PATH . '/configs/application.ini');
Zend_Registry::set('google',    new Zend_Config_Ini(CONFIG_PATH, 'google'));

This will add your api key to the registry so that it can be accessed throughout your application.

Model

In your models directory, create a new class called ‘Geocoder.php’. This will be the file that contains the application logic for retreiving coordinates on a given location.

models/Geocoder.php

/**
* Geocoder
*
* @author Robert McVey
*/
class Geocoder {
	protected $key = "";
	public function __construct($apiKey)
	{
		$this->key = $apiKey;
	}
	private function _getGeocodedLatitudeAndLongitude($address)
	{
		$client = new Zend_Http_Client();
		$client->setUri($this->getGeocodingUri());
		$client->setParameterGet('q', urlencode($address))
			->setParameterGet('output', 'json')
			->setParameterGet('sensor', 'false')
			->setParameterGet('key', $this->key);
		$result = $client->request('GET');
		$response = Zend_Json_Decoder::decode(
			$result->getBody(),
			Zend_Json::TYPE_OBJECT
		);
		return $response;
	}
	public function getCoordinates($address)
	{
		$response = $this->_getGeocodedLatitudeAndLongitude($address);
		$return = array();
		if(isset($result->Placemark[0]->Point->coordinates[1])){
			$return(
			'lat' => $result->Placemark[0]->Point->coordinates[1];
			'lon' => $result->Placemark[0]->Point->coordinates[0];
			);
		}else{
			$return = null;
		}
		return $return;
	}
	private function getGeocodingUri()
	{
		return 'http://maps.google.com/maps/geo';
	}
}
?>

Usage

Alright, that was pretty easy. Now we just have to implement this in our controller. Here is a sample model that uses the Geocoder class to return geotagged User objects.
models/User.php

class User extends Zend_Db_Table_Abstract{
	protected $_name = 'users';
	protected $key;
	protected $geocoder;

	public function init()
	{
		$this->key = Zend_Registry::get('google')->google_map_key;
		$this->geocoder = new Geocoder($this->key);

	}
	public function getUsersAndGeocode()
	{
		$result = $this->fetchAll();

		$users    = $result->_toArray();
		foreach($users as $user)
		{
			$address = "{$user['address']} {$user['city']} {$user['state']} {$user['zip']}";

			$latlon = $this->geocoder->getCoordinates($address);
			if($latlon)
			{
				$user['lat'] = $latlon['lat'];

				$user['lon'] = $latlon['lon'];
			}
		}
		return $users;
	}

}
?>

The controller part is a little too easy. We will setup a reference to our user table and create a map action. Within the map action we get all of users and geocode them. We pass that collection to our view. This file should be located here:
controllers/UserController.php

class UserController extends Zend_Controller_Action{
	protected $user_table;
	public function init()
	{
		//database connectivity stuff is up to you

		$this->user_table = new User();
	}
	public function mapAction()
	{

		$users = $this->user_table->getUsersAndGeocode();
		$this->view->users = $users;
	}
}
?>

And finally, the view. Here we will create our map div, load the Google Maps API (don’t forget to sub out your key in the script src). We are using a custom marker here (personIcon); you can add your own image here by changing the path to your image. I recommend the image be pretty small with a transparent background. Given our current setup, this file would be located at this path:
views/scripts/user/map.phtml

<div id="map">

<noscript>
<center>
<strong>To view the map feature, you need to have Javascript enabled. If you are unsure about how to turn Javascript support on, please</p>
<p>read Google's instructions found here: <a href="https://www.google.com/support/adsense/bin/answer.py?hl=en&amp;answer=12654" target="_blank">LINK</a></strong>

</center></p>
<p></noscript>
</div>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>

<!--
<script type="text/javascript" src="http://maps.google.com/maps?file=api&amp;amp;v=2&amp;amp;sensor=false&amp;amp;key=your_key"></script>
-->

<script type="text/javascript">
 //Sorry, I love jquery! If you don't use window.onload = function(){initialize();}
    $(document).ready(function(){
        initialize();

    });
var map;
var personIcon;
function initialize() {
    if(GBrowserIsCompatible){

        map = new google.maps.Map2(document.getElementById("thurtene_map"));
        map.setUIToDefault();
        map.setCenter(new google.maps.LatLng('48.858001709', '2.29460000992'), 4);
        <? endif; ?>

        personIcon            = new GIcon();
  /**
  * Change this marker to be a small transparent custom image
  *
  */

        personIcon.image      = '/img/common/custom_marker.png';
        personIcon.shadow     = "http://www.google.com/mapfiles/shadow50.png";
        personIcon.iconSize   = new GSize(32,32);
        personIcon.shadowSize = new GSize(37,34);

        personIcon.iconAnchor = new GPoint(9,34);
        personIcon.infoWindowAnchor = new GPoint(9,2);
    }
    addOverlays(map);
}

function personMarker(point, index, dto){
 //set our marker to be the custom icon we created
    markerOptions = {icon:personIcon};
    var marker = new GMarker(point, markerOptions);

/**
 * This is the content in the info bubble
 */
    GEvent.addListener(marker, 'click', function(){
       marker.openInfoWindowHtml(
            '<h3><a href="/profile/view/'+dto.user_id+'">'+dto.user_first+' '+ dto.user_last+'</a></h3>
'
            +dto.user_address+'

'
            +dto.user_city+'
'
            +dto.user_state+' '+dto.user_zip
        );

    });
    return marker;
}
/*
* Here is where we loop through our users and create new markers for each

*/
function addOverlays(map){
    <?php
  $geo = $this->users->_toArray();
        $size = sizeof($geo);

 ?>
    <? for($i = 0; $i < $size; $i++): ?>
                var latLon = new GLatLng(<?php echo $geo[$i]['lat'];?>, <?php echo $geo[$i]['lon'];?>);

                map.addOverlay(personMarker(latLon, <?=$i?>, <?=Zend_Json::encode($geo[$i])?>));
    <?php endfor; ?>
}
</script>

Post a comment if you have any questions.

Read More

Fix 1: Get the computer to boot:

Does the computer boot? If not, what messages (if any) are you getting? If you are getting error messages, Google them; most likely, you are not the first person to experience this problem. Careful where you click, though. Assuming you have eliminated the possibility of hardware failure (and that you have properly backed your important documents up): if the computer doesn’t boot, you could have a boot sector virus. To get rid of these, load the Windows CD and press ‘R’ when prompted to enter the Recovery Console. Type these commands:

• fixmbr
• bootcfg /list
  • If no entries are listed, type bootcfg /rebuild
  • Enter the numerical identifier for your Windows installation (likely the number 1)
  • Type Y or Yes to add installation to boot list
  • Provide a load identifier (e.g. Windows XP)
  • Enter /fastdetect

Fix 2: Finding the culprits:

If it’s possible, hook the infected hard drive up to a different machine and scan it using MalwareBytes, removing any infected objects it finds. Now, knowledge of an approximate time the virus was contracted is REALLY helpful here, though not necessary. If you can’t hook it up to another computer, boot the computer into safe mode (press F8 as the computer is booting) and choose Safe Mode (with Networking).

Whether you have booted to this hard drive or are viewing it from a different computer, take the following actions:

Open My Computer > C:. Go to Tools > Folder Options > View tab > Check “Show hidden files and folders”; uncheck “Hide extensions for known file types”, “Hide protected operating system files” and “Use simple file sharing”. > Change to detail view and sort by Date Modified, more than likely, all of the virus files are going to have the same date in this field. Now is the tedious process of looking (and deleting/renaming), be sure you aren’t deleting important system files by using Google. Here are the important places to look, though they can be anywhere:

• C:\
• C:\WINDOWS\
• C:\Recycler\S-1-{RANDOM}\
• C:\WINDOWS\Tasks
• C:\WINDOWS\system32\
• C:\WINDOWS\system32\drivers\
• C:\Program Files\{ANY RECENT SOFTWARE THAT IS NEW AN YOU DIDN’T CHOOSE TO INSTALL}
• %TEMP% (Start > Run > %TEMP% > OK)
• %USERPROFILE%\Desktop (look for installer files)
• “%USERPROFILE%\Local Settings\Temporary Internet Files\”

Open Windows Firewall through Start > Control Panel > Windows Firewall, click on the Exceptions tab and check for any programs that you didn’t specifically authorize, remove the exception if there are unknown entries.

Fix 3: Stop the rogue processes from loading at startup

Download Autoruns from Sysinternals. Run the program and select the Logon tab. Check for malicious software under the headings listed below, unchecking each malicious item:

• HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
• HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
• C:\Documents and Settings\All Users\Start Menu\Programs\Startup
• C:\Documents and Settings\username\Start Menu\Programs\Startup

Move to the Services tab and look for items where the Publisher is missing, these items are frequently (not always) suspect. Remove malicious services by unchecking them

Select the “Image Hijacks” tab, the only item that should be present is “Your Image File Name Here without a path”. If anything else is present, uncheck it.

Close Autoruns

Cleanup

Download CCleaner, install and run it with the default settings to remove all of your temporary files. If you don’t have a good Firewall and don’t have money to spend, download Comodo or ZoneAlarm. Turn off System Restore by right-clicking My Computer and selecting Properties. Select the System Restore tab and check the Turn off System Restore checkbox > Select Apply (may take a moment or two). Once it is responding again, uncheck the box and it will create a new restore point (that doesn’t have the virus files).

Some of the other benefits of this free program are:

• Search by file type
• Search specific drives
• Hidden system files
• Securely overwritten files
• Zero byte files
• It’s Free!

1. Record the podcasts
2. Create an XML file that describes the content (author, file location, etc.)
3. Upload the audio and XML files to a web server

The unlisted step 4 is share the link to your feed (XML file) with friends/colleagues/customers. That is really all there is to it. If you want iTunes to follow you, of course there is a little more work that goes into that.

If that seems straightforward, except for the XML part, I have generated a little tool to help you on your way. It won’t record the podcast for you or make you like the sound of your own voice, but it will generate the XML file you need.

Podcast XML Generator

http://mediamemo.allthingsd.com/20090913/home-delivery-the-new-york-times-serves-up-some-malware/

fixtm.reg


REGEDIT4

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\System]
"DisableTaskMgr"=-

FixExe.reg


REGEDIT4

[HKEY_CLASSES_ROOT\exefile\shell\open\command]
@="\"%1\" %*"

Batch file

<@echo off
:: SET_NO_DRIVE_OTORUN
reg add HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer /v NoDriveTypeAutoRun /t REG_DWORD /d 0x0ff /f

:: GET_DRIVES
for /f "tokens=1 delims=:" %%j in ('reg query hklm\system\mounteddevices ^| findstr \DosDevices\') do (
echo %%j >> drives
)

:: REMOVE_\DosDevices\_PREFIX
for /f "tokens=3 delims=\" %%j in (drives) do (
echo %%j >> drives.txt
)
del /q /f drives > nul

:: REMOVE_SPACE
for /f "tokens=1 delims= " %%j in (drives.txt) do (
echo %%j: >> drives
)
del /q /f drives.txt > nul

:: CHECK_DRIVE_TYPE
for /f %%j in (drives) do (
fsutil fsinfo drivetype %%j | findstr "Fixed " >> fdtype
fsutil fsinfo drivetype %%j | findstr "Removable " >> frtype
)
del /q /f drives > nul

:: GET_FDRIVES
for /f "tokens=1* delims= " %%j in (fdtype frtype) do (
echo %%j >> dtype
)
del /q /f fdtype > nul
del /q /f frtype > nul

:: REMOVE_SPACE1
for /f "tokens=1 delims= " %%j in (dtype) do (
echo %%j >> drives
)
del /q /f dtype > nul

:: DEL_DRIVE_A_FROM_LIST
sort drives >> sort
type sort | findstr "A" > nul
if errorlevel 0 for /f "tokens=1 skip=1" %%j in (sort) do (
echo %%j >> sorted
)
del /q /f drives > nul
del /q /f sort > nul

:: CREATE_OTORUN_FOLDER
for /f %%j in (sorted) do (
md %%j\AUTORUN.INF
attrib +s +h +r /d /s %%j\AUTORUN.INF
)
del /q /f sorted > nul

echo Press any key to close this window..
pause > nul>

Here’s what you do:

• Go to http://skydrive.live.com and either sign-in or sign-up
• Upload a Word/Excel/PowerPoint 2007 document. Select the file you uploaded, click on “Edit” (at the time of this writing, Word was not available)
• Download Gladinet from here: http://www.gladinet.com/p/download_starter_direct.htm
• Install the free version
• Mount a network drive, select Windows SkyDrive and enter your Live credentials
• Use Comodo backup or SyncBack to setup backups to the newly mounted network drive
  • Weekly/monthly backups may be best as Gladinet limits the number of items that can be transferred per month in the free version.

More information from Symantec (source of the fix) here: http://www.symantec.com/security_response/writeup.jsp?docid=2004-050614-0532-99