Custom rendering of FormCollection Elements

A problem with ZF2 forms is that you can’t actually configure the rendering of a form element. Same (stupid) story that was in ZF1 it’s in ZF2 as well.  If you want to change the rendering the only (idiot) solution is to extend the original rendering helper and that means override the render() function, put the whole code from the original render() function ( and edit it with your changes). The issue is you copy too much code from the framework itself and you should maintain it.

Problem: need to modify the rendering to adapt the HTML to be compatible for Bootstrap3.
Solution:

Extend FormCollection and FormRow. Only one thing to change for the extended FormCollection, just set the $defaultElementHelper:

<?php
namespace ApplicationFormViewHelper;

use ZendFormViewHelperFormCollection;

class FieldCollection extends FormCollection
{
    protected $defaultElementHelper = 'fieldRow';

}

The new FieldCollection will use the view helper called fieldRow:

<?php
namespace ApplicationFormViewHelper;

use ZendFormViewHelperFormRow;
use ZendFormElementInterface;

class FieldRow extends FormRow
{

public function render(ElementInterface $element)
    {
   //.... copy code from FormRow and modify with your changes
  }
}

To be able to call the new helpers, you need to register then as invokables in Module.php

  public function getViewHelperConfig()
    {
        return array(
            'invokables' => array(
                'fieldCollection' => 'ApplicationFormViewHelperFieldCollection',
                'fieldRow' => 'ApplicationFormViewHelperFieldRow'
            ),
           ....
}

My changes were mostly where the markup is created, so in my new FormRow renderer I have something like:

 $markup = '<div class="form-group">'. $labelOpen . $label. $labelClose .
                                    '<div class="col-sm-5"><div class="input-group">'.$elementString .'</div>'.'</div>'.'</div>' ;

and I also set a default class for the label element

 $labelAttributes = array('class'=>'control-label  col-sm-5');

Now in the HTML template we just use

<?php
    echo $this->fieldCollection($formElement);
?>

Word boosting in Sphinx

disclaimer : I’m not an expert in Sphinx

Boosting words

One feature that Sphinx is missing , but it’s found in Solr/Lucene is word boosting – a.k.a you want some words which if are found, the document gets a weight boost . in Solr/Lucene you can define this by using ^ operator , like word^x , where x is a number and a multiplicator for that word’s score .

How to do that in Sphinx ? Well , it’s another play with the query string . Please note that the procedure will not actually give you the word boosting found on Solr ,it’s more a workaround .

Let’s assume you have a title and a description that are indexed in sphinx .  Let’s take an example “techno music” . You will search this string in both title and description ( you might have different weight set for each field , in most cases title gets a field weight boost ) .  Let’s say you have a query like “^techno music$”|”techno music”|”techno music”~10 to have a general good search ( first is exact match on a field , second is exact phase , third is a 10 words promixity match ) . The reason you want this boosting is mainly because your titles might not have both words and you are intested of those who have techno word –  since music is more general . To boost techno you need to add another |”techno”/1 . What you will achieve is , beside the phase search that should match content , you are matching ( if exist ) the word tehno in the title . It could also match only techno in description , but since you consider it an important term , then it’s no problem , you achieved the desired search. If it’s only one word , then you don’t need the /1 operator . But if you would have techno house as boosted words , your extra query would be |”techno house”/1 , to give a match of at least one .

The list of boosted words can be held in a file which is readed when doing a search . You can also use a memory caching ( memcached , apc ,whatever ) .

UnBoosting words

What happends if you want the opposite ? Let’s take the above example : techno music . Music is a general term . You could implement the above procedure and make a list with music subgenres and boost them , but why not decrease the score for music term ? Sphinx has the stopwords list feature , but if you add music there , it won’t be counted in search . And you don’t want that , because you might want to match music too , if techno is not found . The idea is you want first the docs that match techno and then those that match music .

The implementation is similar with the one for boosting :

  • read the list of words that are not important
  • create a new string that contains the input string but without the words in the list above ( use str_replace to delete them)
  • to your query search add a |”new_string”/1

What we have now it will be existing_query | “new_string”/1 , so we basicly made the new_string to be a boosted list of words . So if you searched for “techno house music” and music is in your not important words , the query will be now “techno house music” | “techno house”/1 . First results will contain techno,house and music , then techno OR house . If you use a more relaxed query like |”techno house music”/10 , then you might get more results  ( with less relevance ) .

Both procedures will help your results especially when you are searching in multiple fields with variable weight . In both cases you will do an OR which search only for some words ( which are considered more important ) in case they can match in smaller fields ( like a title ) . Of course , you could do | @title term  and in this case you are making a match if exist only in that field .

Happy Sphixing 🙂

Synonyms in Sphinx

disclaimer : I’m not an expert in Sphinx

Sometimes when doing a search you want to search not only for the words included in the query , but also after their synonyms , to increase the number of results . Sphinx doesn’t come with this by default . Instead it comes with a feature names “wordforms” . Yet wordforms is not a fully featured synonyms feature . As it’s name , it take cares of forms ( variations) of a word , mispells OR direct several uncommon words to a single one . Bear in mind : a single one. You can’t declare dog > cat and then make cat > dog . So you could do dog > cat and mouse > cat , both will be replaced by cat when searching , but you can’t make to search for all 3.

So , how to do it? Since we don’t have  a feature implemented , only options left is to use the query string  : let’s say we search for “black cat” and we have for cat the synonym dog . Our query will transform from “black cat” into “black cat|dog”. Sphinx will return both “black cat” and “black dog” matches.

How to do that :

– first we create a file ( let’s say it synonyms.txt ) in which we put on every line a synonyms list

– when we receive a query string , we take the query string , split it in words and for every word we search in this file for a match

– match found , we modify the query string to by replacing the word with the words found in that line

– do the search

Problems :

– obviously , response time always grows with the length of the query search ( and filtering etc. ) . This new query with OR operators shouldn’t increase very much the response time ( well , you might notice it if your collection of data is big and you get a lot of traffic )

– searching for the synonyms . Here you could get trouble , especially if the file grows. The simple way is to read each line , explode it and search for every word if is in the array.  This is not very efficient and for a big file this is a problem , since you will consume a lot of memory. Alternatives might be :

  • use grep – it’s pretty fast and will return you the matched line ;
  • use memcached for a matched line of a certain word . You can store a key like ‘synonym_cat’ with content ‘cat|dog’ .  Best would be to have memcached on the same server , to avoid network lagging  .
  • use APC instead of memcached . You could also cache the function that does the search in the file

Here is an example on how to do ( please note that this solution is not optimal for large files ):

Each line of synonyms.txt will look like this :

cat | mouse | dog

If you use another separator , be carefull to replace it with | ( OR operator) when inserting in the query string .

$lines= array();
$synofile = file("synonyms.txt");
foreach($synofile as $line){
   $lines[] = trim($line);
}
$tmp_string = strtolower(str_replace(array('-','+'), " ",$input_string));
foreach ($tmp_string as $word){
  $extraword =false;
  foreach ($lines as $line){
    if(false !==strpos($line,$word)){
       $input_string= str_replace($word, $line, $input_string);
    }
  }
}

As I said , this is not a perfect solution , for example it should test the words for a minimum length .

Personal project : Huzmet.ro

My personal project enters in a pre-beta phase . Currently the design is not ready yet , so it’s more about functionality .
Some tehnical details : made on Zend Framework , uses SphinxSE for searching . There’s no caching used yet , but will use ( most likely) memcached and file caching ( using Zend Cache ). There’s also a plan for using Gearman workers .
The project will start first for Romania , but multi-language is already implementend ( get-text translations) .
Testing is welcomed , especially testing with romanian texts/queries , but english is fine too .
Huzmet is a local services providers directory and more . And by that more one feature is that instead of searching you can make a shout , like a request , the shout is scanned and if matches are found among the providers , they get informed about your shout and they can contact you .
Huzmet.ro

non-input Zend Form Element

Simple case : let’s say you make a form for user to register . You have an email field , right ? Ok , now when the user is logged you have a page where he can modify password and other settings . You should normally want to display the email too , but you don’t want to be editable ( because emails should unique etc. ) .You could show the email separated from the form , but you don’t want that , you want it to be in the same area with the other fields .
One simple approach is to create a “dummy” element , one that can be populated by the populate() function , but actually don’t have an <input> . The way is to create an element that extends Zend_Form_Element or Zend_Form_Element_Xhtml which uses formNote helper . Normally this helper is used by hidden elements ( it’s a dead simple helper , you can check the source in Zend/View/Helper) , but you can use it in this case too :

class App_Form_Element_Xhtml extends Zend_Form_Element_Xhtml
{
public $helper = 'formNote';
}

Now ,you will proubably extend your form when editing ( I use a Base form with the fields and for Edit/Register I extend it with new ones ) . To get the email displayed , you need first to remove the email element ( text type ) and replace it with this one :

$this->removeElement('email');
$fakeEmail = new App_Form_Element_Xhtml('email',array(
                 'required' => false,
                'ignore'=> true,
		'label' => 'Email : ',
		));
$this->addElement($fakeEmail);

You need to add ignore to be true , otherwise $form->getValues() will return you a “” value form the email – this can be fixed by declaring getValue and setValue functions for the element , but setting ignore works too .
The last step is to re-order . If the email is the first you can use $fakeEmail->setOrder(-1) before adding .

Eclipse , xdebug , remote system

This morning I had nothing better to do ( actually I had , but … ) so I thought “let’s try xdebug” . I’m working remote on a EC2 instance using Eclipse PDT . Installing xdebug is pretty easy , apt-get install php5-xdebug or something like that ( on debians) , you need to edit a .ini file then you create a debug profile in eclipse ( bla bla bla there’re a lot of tutorials about it ) .

Hit debug buton annnndd ….. “launching: waiting for xdebug session” . Wtf … doesn’t work .

Gogled a bit more , found that you should have the xdebug option xdebug.idekey to be the same with XDEBUG_SESSION_START used by the IDE . IN this case it’s ECLIPSE_DBGP . Netbeans eg. have netbeans-xdebug I think . Well, you can set it to whatever .

Hit button annndd … “launching: waiting for xdebug session” . Wtf … doesn’t work .

I remember I used xdebug some time ago , but I was running locally . So what could it be ? Well xdebug to help you need a remote address . By default is localhost . If you work remote , you should have instead of localhost your IP . Well , if you have a public IP , lucky you , but I bet you don’t have .

What to do ? SSH tunnel . You can do it . OR you could use the built-in option PDT have 🙂

Zend Forms

Zend Forms are a nice tool in ZF , but it’s a bit weird and looks bloated for new people . One thing is that it follows the decorator pattern and another is that you end up writing some code for a damn html form of several lines . Yet , it offers a validation system and if you have many forms in your project , the decorations might turn in a not bad thing .
After reading the tutorial , first thing you will want to do is to get rid of the “damn” dt/dd . The dt/dd combination was chosen to be default for a number of reasons ( some good ones ) . Yet designers might look weird at you ( even if dt/dd can be handled pretty well ) or you just want div tags or a damn table there .
How to do that ? well ,there are 2 ( no , 3) sollutions : one is to modify the decorators , second is to create your own elements ( maybe with their decorators too) … and last one is to use the ViewScript decorator ( basicly you template the form ) . The second , even if looks nice – to create your own element , sucks because you need to re-create all the input elements . And most of all sucks because you re-invent the wheel . Writing a decorator that pretty much does same thing as the default ones , sucks too.
ViewScript is a nice sollution , the problem I see here is that if you want to change your forms layout , you need to edit all those template files ( or maybe I’m wrong ). It’s a good sollution , but I want to use the normal path , if those zf people made the effort to build it .
So … after googling and googling , I have to say the docs are pretty bad . Even tutorials . They don’t explain exactly how to “reset” the damn decorating .
Let’s take it easly and explain how decorators apply ( and how forms works ) . It’s easy : you give an array which is processed … in the order you gave it . First decorator will be always the ViewHelper . This one renders your <input> element and nothing else ( along with the attributes you set for it ). The next decorator you declare will “embrace” the content you already have . If you have a label , it will be concated with the <input> element . If you have a HtmlTag , it will include your input or whatever is the current content. If you have another Html, it will embrace the new content , -that’s your previous HtmlTag which has the input side .  And so on . It’s like having boxes of different sizes in the order of first is the smallest , last is the biggest .  Pretty much same thing is when you add things in a form class , either they are elements or groups or decorations . There is one thing : you can define order here ( I’ll show later an example).

So , let’s say we want to have table instead of dt/dd .

$this->addElement('text','name',array(
			'filters' => array('StringTrim'),
			'validators' => array(array('StringLength',true,array(3,128))),
			'required' => true,
			'label' => 'Name :',
			'decorators' => array('ViewHelper',
						array(array('linebr' => 'HtmlTag'),'options'=> array('tag' => 'br', 'placement' => 'append','openOnly'=>true)),
						array('Errors'),
						array(array('data'=>'HtmlTag'),'options'=>array('tag'=>'td')),
						array('Label','options'=>array('tag'=>'td')),
						array(array('row'=>'HtmlTag'),'options'=>array('tag'=>'tr')))
			));	

This will render something like this :

<tr>
    <td>
        <label>Name:</label>
    </td>
    <td>
        <input type="text" name="name" id="name" value="">
       <br/>
       <ul><li>Error 1</li></ul>
    </td>
</tr>

Let’s recap : first we have the input element , rendered by ViewHelper , then we have a br line ( use openOnly to not create 2 br’s) – also note we use placement ( if we use prepend , the br will be pasted before the input ) , then we have Errors . Next comes a td tag which will embrace everything we have until now . Now it’s the Label time , instead of dt we have a td . By default it’s prepended . The last one is the tr tag that completes our row . Note that for multiple HtmlTag decorators you need to give them a key(or name , whatever).This is to create a new HtmlTag instance , otherwise the same decorator is applied – you can try removing the keys to see what will happen .
This looks ugly … if you do it for all your elements . But we’re in OO world so we can improve it. But first , let’s say you add several more elements .At the end you will add this :

$this->setDecorators(array('FormElements',array('HtmlTag', array('tag' => 'table')),'Form'));

This will render the table tags . Remember this must be added after you inserted all your elements , more likely after the submit button .
Ok , how can you make this spagetti be more nicer ? Your could use overwriting the default decoratos for elements , but I’m using another alternative : simply create a form with no elements , you create a method that returns the whole decorator array for the elements and you can have something like this :

$this->addElement('text','name',array(
	'filters' => array('StringTrim'),
	'validators' => array(array('StringLength',true,array(3,128))),
	'required' => true,
	'label' => 'Name :',
	'decorators' => $this->decorators()
));	

Looks a bit nicer now . Even more you can have a method :

public function loadDefaultDecorators()
{
      $this->setDecorators(array('FormElements',array('HtmlTag', array('tag' => 'table')),'Form'));
}

This is be the default decorator for the form , so you don’t need to add it in init() . Your form will extends this one instead of default Zend_Form .
So now you can create forms that extends this one and you will get html table output .
It’s very likely that you’ll have several types of decorators , one is for sure the submit button – because you don’t have a label there .

public function submitdecorator()
{
	return array('ViewHelper',
			array(array('data'=>'HtmlTag'),'options'=>array('tag'=>'td')),
			array(array('data2'=>'HtmlTag'),'options'=>array('tag'=>'td','placement'=>"prepend")),
			array(array('row'=>'HtmlTag'),'options'=>array('tag'=>'tr')));
}

Here you go . For submit button you do ‘decorators’ => $this->submitdecorator() . You can have a label too , but it will be ignored , because we didn’t defined the decorator for it . Note that for the second HtmlTag I used prepends , otherwise the cell with the submit button will be in left and we want it in right (to be aligned with the inputs , not the labels , but as you wish ) .

Ok, what’s next : using DisplayGroup . Let’s say you have you form table with several input elements , but you also have 2 checkboxes that you want to be displayed in the same cell .
First , you can’t use the decorator above , because it will render one element per row . Instead you want the 2 checkboxes to be rendered inside the cell and every checkbox to be inside a div for futher styling .

'decorators' => array(
			'ViewHelper',
			array('Label','options'=>array('placement'=>'append')),
			array('HtmlTag','options'=>array('tag'=>'div'))

This will be your decorator for each checkbox . Note that I put the label after the checkbox .
Next is to group them :

$this->addDisplayGroup(array('firstcheckbox','secondcheckbox'), 'nameofgroup',
		array(
			'description' => 'Bla bla bla :',
			'decorators'=>array('FormElements',
					  array(array('data'=>'HtmlTag'),'options'=>array('tag'=>'td')),
					  array('Description','options'=>array('tag'=>'td','placement'=>"prepend")),
				          array(array('row'=>'HtmlTag'),'options'=>array('tag'=>'tr'))
		)));		

‘firstcheckbox’ and ‘secondcheckbox’ are the names of your checkboxes .You always need to add there the elements names , otherwise they are not included in the group . I used Description for filling the left row .

Last thing : I said something about order of elements . How is useful . Let’s say you made a form … user data or something . Somewhere you have an admin are where as admin you can edit this data . Most likely you wil have several additions fields ( like if user is banned or something ) . You can extend the form you have . The problem is when you add new fields , they will be added AFTER the submit button . Why ? because elements are rendered in the order they are added , except if you set an order . By default , there is no order , it’s the normal iteration of the array with elements . But you can set it . The most simple way is to set a high value to the submit button – it will be rendered the last one . You can do it even in your admin form :

$this->getElement('mysubmit')->setOrder(100);

This way you don’t need to touch the initial form .

nginx headers error

If you get this error in nginx :

upstream sent too big header while reading response header from upstream

and your upstream is a lamp server ( possibly for others too) , the problem might be that you are sending a header that is too big for the current settings of nginx .

My problem was that I was setting a lot of cookies . I had output_buffering enabled in php and set to 4096 . Nginx was configuring with default proxy-ing values . After I digged a bit , I solved this by modifying output_buffering On in php . When is On , the header is sent when it’s finished and not in chunks (at least this is what I understand from php doc) . I tried with 9192 value , but same thing .

In nginx I was using default  proxy configuration , with proxy_buffering on . After googling a bit , I found out some guys that had a simillar problem with fastcgi . The sollution was to increase the buffers sizes :

proxy_connect_timeout 90;
proxy_send_timeout 180;
proxy_read_timeout 180;
proxy_buffer_size 16k;
proxy_buffers 8 16k;
proxy_busy_buffers_size 32k;
proxy_intercept_errors on;

Proxy_intercept_error is optional . Also you can play with there values . Using higher values only makes nginx to use more memory . If you have plenty memory , this shouldn’t a problem  🙂

Writing logs with ZendLog

Sometimes is a web application is demanded to have logging for purposes like debugging or user-actions logs .
There are many sollutions for adding logging abilities to your site , starting with the built-in php functions for writing in log files and ending with special libraries of this purpose . One of them is ZendLog library from Zend Framework.
One of the advantages of ZendLog is that it enables to define additional levels of logging , beside the ones that are embedded in the class.
In is also usefull that it supports multiple file writing , for example you can write error-level logs in a file and debug-level logs in another file , or having an user-action logging in a separate file from the debug logs.
In this example we’ll create a class that will extend ZendLog and add several usefull capabilities.
For the start , we’ll include the ZendLog files:

require_once ‘Zend/Log.php’;
require_once ‘Zend/Log/Writer/Stream.php’;
require_once ‘Zend/Log/Filter/Suppress.php’;

You can change the paths depending where your ZendFr classes are located.
Now we create our class:

class Logger extends Zend_log{

First method that we’ll define it , it will be init() . We’ll assume we have some global array which will contain files paths are several settings . Those can be modified according to you envoirement .

private function init(){
global $conf_log_settings;

Lets assume we want a user-actions loggins , we can name this level USERACT and give it a 8 priority number

$this->addPriority(‘REPORT’, 8);

Now we start to define the first streaming for debugging logs . We give the path of the log file and also we define a custom formatting :

$debug_writer = new Zend_Log_Writer_Stream($conf_log_settings[‘debug_logfile’]);
$formatter = new Zend_Log_Formatter_Simple(‘%timestamp%|%priorityName%|(%priority%)|%message%’ . PHP_EOL);
$debug_writer->setFormatter($formatter);

We can define a surpress filter if we want for example to disable the logging in the production envoirenment

$bansher = new Zend_Log_Filter_Suppress();
$bansher->suppress($conf_log_settings[‘debug_enable’]);
$debug_writer->addFilter($bansher);

To disable the debug loggigng ,$conf_log_settings[‘debug_enable’] must be set to TRUE .
Now we can define a level of debugging , we can let all debug messages or limit only to Emergency level . ZendLog levels start from 0 to 7 as priority. So setting to 7 will allow all messages to be written .

$filter = new Zend_Log_Filter_Priority($conf_log_settings[‘debug_level’],’<='); $debug_writer->addFilter($filter);
$this->addWriter($debug_writer);

And we are done with the debugging messages . Now we can do the separate logging for our user-actions . We’ll need to define a new stream :

$action_writer = new Zend_Log_Writer_Stream($conf_log_settings[‘action_logfile’]);
$formatter2 = new Zend_Log_Formatter_Simple(‘%timestamp%|%priorityName%|(%priority%)|%message%’ . PHP_EOL);
$action_writer->setFormatter($formatter2);

We set set same costum formatting as we had for debugging , but we could eliminate for example the priorityName , because our file we’ll have only one type of messages.
Next is to tell ZendLog to log on this stream only the USERACT level messages ( which we defined earlier to have priority 8 )

$filter2 = new Zend_Log_Filter_Priority(8,’=’);
$action_writer->addFilter($filter2);
$this->addWriter($action_writer);

And we are done with the init function .
Because we’ll use this class in various places and to not initiate all the time and run init() function , we can use a Singleton pattern to ensure init() will be called only one time . This is done with the classical getInstance() function , except we’ll run init() also in it :

public static function getInstance()
{
if (self::$instance == NULL) {
self::$instance = new Logger();
self::$instance->init();
}

return self::$instance;
}

Last function that we’ll define is a nice introduction in php5 : the __call function . __call function role is to catch situations when we call a method which is not defined . ZendLog use this function to create a method for each of the defined levels of logging . We’ll extend ZendLog __call to add an improvment

public function __call($method, $params){
$param=””;
foreach ($params as $par){$param.=”$par|”;}
$param = rtrim($param,’|’);
parent::__call($method,array($param));
}

Normally ZendLog __call reveices an array of araguments . Our modification is to add a | between the parameters , because we used a custom formatting to the line . Otherwise ZendLog will use it’s separator , but we want the message to look unitary . We’ll explain this in how to use our class.
To write a log line we need to make a call like this :

Logger::getInstance()->debug($a_message,$an_user_id);

The line written in the log will be in the following format :

2007-11-05T18:57:59-05:00|DEBUG|(7)|Hello world|14

where “Hello world” is the message and 14 it’s an user id .
The good thing about our __call is that we can add how many parameters we want , and those will be inserted in the log file.
If we are doing debbuging sometimes it is usefull to know where the log was made . For this we can include the magic constants __LINE__ and __FILE__
__FILE__ will add the file were the log was made and __LINE__ will give us the line number in that file

Logger::getInstance()->err(__FILE__,__LINE_$a_message,$an_user_id);

The log line will look like :

2007-11-05T18:57:59-05:00|DEBUG|(7)|helloworld.php|10|Hello world|14

So Logger can be called with debug(), err(), warn() . These are the name of the priorities . See inside ZendLog class for the whole list from 0 to 7.
Finally , let’s show an example for our user-actions logging :

Logger::getInstance()->useract(‘user logging’,$uid);

The line in the log file for those actions will look like :

2007-11-05T18:57:59-05:00|USERACT|(8)|user logging|13

I hope this class can help on tracking debugging , user-actions or any events that you need to track .
Many thanks to Dario Sande for reminding me about the magic constants __FILE__ and __LINE__ . An implementation of this class can be done for CodeIgniter framework (thanks Fernando Varesi ) .
The only difference is that you don’t need anymore getInstance() , because CodeIgniter implements the Singleton , and you need to declare the contructor as

function __construct() {
parent::__construct();
$this->init();
}

mEgo launches !

Yes , we are in beta . The whole team worked very hard to get here are we are pleased of the result . Yesterday we had our presentation at the TechCrunch40 conference and we did very well. The audience liked us . Curently the beta is limited , soon we will release progressivly invitations to the people that had already subscribed to our mailing list .
Yesterday we had our presentation at TechCrunch20 (actually they were 40 ) . We didn’t win , but we made a good impression to the audience . The way is still long , we need to see how our servers will handle more and more users . We are testing and testing the current features .We want to have a very stable site 😉