Creating custom Magento Index

Jan 3, 2014

by Mexbs team

Magento Tutorials

In this tutorial we will explain about the indexes in Magento, their usage and design. Afterwards we will show you how to create and configure your own index.

The usage of indexes in Magento

Roughly saying, indexes in Magento are used to maximize the website performance on the frontend, by performing all the resource-consuming calculations on the backend, caching them (mostly to the database, but not necesserely) and using the cached data in the frontend.

Example of usage of Indexes in Magento

Let's take a look on the following example: Let's imagine that there is a registered customer browsing in your Magento catalog. This means that you need to display the products prices. And imagine that you have a catalog rule, which applies 20% discount for registered customers only on some chosen products.

Let's see what happens if we don't use indexes: Each time when the catalog is displayed on the frontend, magento needs to get the customer type, to get the product and to try to apply our catalog rule in order to get the correct final price to display to the customer. This approach has two obvious problems:

The frontend is getting much slower due to the calculations which are done for every single product which price needs to be displayed.
Exactly the same catalog rule calculations are performed lots of times = redundant resources consumption.

It is needless to say that those two problems are a killing factor for ecommerce website.

Let's see what happens if we use indexes: First, in the moment that you apply the catalog rule, Magento empties the "catalog_product_index_price" table (and other price index tables, depends on the product types), calculates the new prices for each product and each customer group, and inserts this fresh data to "catalog_product_index_price" table. This process is called reindexing. Additionally, each time a product price is modified, product is deleted, customer group is modified or deleted, the reindexing is performed. Thus, thanks to the reindexing, most of the time you have a reliable data in "catalog_product_index_price" table, which you can use to display the prices on frontend. The two problems from before are solved! -

There are no more calculations performed on frontend (the prices are fetched from the "catalog_product_index_price" table) = much faster.
There are no more redundant calculations. – The recalculations are done only once when there is a change in the data that influences the prices.

The implementation in Magento

In this section we will explain the implementation and the workflow of the indexing process in Magento. This will help you to better understand how to hook your own index process, which will be described in detail in the next section – "creating custom index in Magento".

The workflow begins when event is logged. The event logging is performed by a call to Mage_Index_Model_Indexer:: logEvent.

Inside the logEvent, the code iterates through all index processes that are configured in the system (including yours, after that you will create it in the next section ;-)). On each iteration, there is a call to matchEvent method of each index process, and if the match is successful (i.e if matchEvent returned "true"), then it calls to the _registerEvent method of the matched process. The _registerEvent method than saves some data to the event object, which it can later use to process the event. Additionally, the logEvent method checks for every process, whether the process mode is "manual", and if so, it marks the process status as "reuqire_reindex" and doesn't call the _registerEvent for this process. In the end, the logEvent saves the event to the database (index_event table).

Here is an example of a call to logEvent (taken from magento ce 1.7.0.2):

class Mage_Catalog_Model_Product extends Mage_Catalog_Model_Abstract
{
…
protected function _beforeDelete()
    {
…
Mage::getSingleton('index/indexer')->logEvent(
            $this, self::ENTITY, Mage_Index_Model_Event::TYPE_DELETE
        );

In this example, the parameters that are passed to logEvent are the product object itself, the entity which in this case is 'catalog_product' and the event type, which is 'delete'.

After the event logging comes the event indexing. This is where the stuff gets done. The event indexing is done by a call to Mage_Index_Model_Indexer::indexEvents method. In indexEvents, the code iterates through all the index processes and for each process it fetches the unprocessed events from the database (index_process_event table). Then it calls again to matchEvent, and checks again whether the process mode is "manual". If there is no match or the process mode is "manual", the method returns, and the index is left unprocessed. Otherwise, the method calls to _processEvent method of the index process. The index process can fetch the data that was saved by _registerEvent, and use it to process the event.

Here is an example of a call to indexEvents (taken from magento ce 1.7.0.2):

class Mage_Catalog_Model_Product extends Mage_Catalog_Model_Abstract
{
…
protected function _afterDeleteCommit()
{
    …
Mage::getSingleton('index/indexer')->indexEvents(
            self::ENTITY, Mage_Index_Model_Event::TYPE_DELETE
        );

To sum up, Magento implements the reindexing by calling to logEvent, and later on in some point to indexEvents. This is the first way of Magento to implement this.

The second way that is used by Magento is by calling to Mage_Index_Model_Indexer:: processEntityAction. This method is logging and indexing the event at once, by calling first to logEvent and then to indexEvent. The difference between this and the first way, is that this method doesn't save the event in the database, thus, it needs to call the indexEvent after the call to logEvent (in order to be able to pass the event object to indexEvent). Pay an attention to the difference between Mage_Index_Model_Indexer::indexEvents and Mage_Index_Model_Indexer::indexEvent. The only significant difference between those two methods is that indexEvents is fetching all the "new" events from the database for every process which it iterates through. indexEvent on the other hand gets the event as a parameter, and trying to process only this particular event for every process.

Creating custom index in Magento

After seeing how the indexes workflow is implemented in Magento, we can proceed to creating your own index process. This is done easily by adding the following lines to the config.xml of your module:

<config>
…
    <global>
    …
        <index>
                    <indexer>
                            <some_unique_identifier>
                                    <model>yourmodule/indexer</model>
                            </some_unique_identifier >
                    </indexer>
                </index>
    …
        </global>
…
</config>

Replace "some_unique_identifier" with a unique identifier for the node. It can be a combination of your module name with some word. It just has to be unique over all xml nodes under the same hierarchy. Mage::getModel("yourmodule/indexer") should get the class which will handle the indexing. In magento this class is called "indexer".

Now create your indexer class. (the one which will be returned in a call to Mage::getModel("yourmodule/indexer")). Copy the following:

<?php
class Yourpackage_Yourmodule_Model_Indexer extends Mage_Index_Model_Indexer_Abstract
{
  /**
     * Data key for matching result to be saved in
     */
    const EVENT_MATCH_RESULT_KEY = 'yourmodulename_match_result';

    /**
     * @var array
     */
    protected $_matchedEntities = array(
        Mage_Catalog_Model_Product::ENTITY => array(
            Mage_Index_Model_Event::TYPE_SAVE,
            Mage_Index_Model_Event::TYPE_MASS_ACTION,
            Mage_Index_Model_Event::TYPE_DELETE
        )
    );

    /**
     * Retrieve Indexer name
     * @return string
     */
    public function getName()
    {
        return 'Your indexer name';
    }

    /**
     * Retrieve Indexer description
     * @return string
     */
    public function getDescription()
    {
        return 'Your indexer description';
    }

    /**
     * Register data required by process in event object
     * @param Mage_Index_Model_Event $event
     */
    protected function _registerEvent(Mage_Index_Model_Event $event)
    {
        $dataObj = $event->getDataObject();
        if($event->getType() == Mage_Index_Model_Event::TYPE_SAVE){
            $event->addNewData('yourmodule_update_product_id', $dataObj->getId());
        }elseif($event->getType() == Mage_Index_Model_Event::TYPE_DELETE){
            $event->addNewData(' yourmodule _delete_product_id', $dataObj->getId());
        }elseif($event->getType() == Mage_Index_Model_Event::TYPE_MASS_ACTION){
            $event->addNewData(' yourmodule _mass_action_product_ids', $dataObj->getProductIds());
        }
    }

    /**
     * Process event
     * @param Mage_Index_Model_Event $event
     */
    protected function _processEvent(Mage_Index_Model_Event $event)
    {
        $data = $event->getNewData();
        if(!empty($data[' yourmodule _update_product_id'])){
            $this->doSomethingOnUpdateEvent(($data['yourmodule _update_product_id']);
        }elseif(!empty($data['yourmodule _delete_product_id'])){
            $this->doSomethingOnDeleteEvent($data['yourmodule _delete_product_id']);
        }elseif(!empty($data[' yourmodule_mass_action_product_ids'])){
             $this->doSomethingOnMassActionEvent($data['yourmodule _mass_action_product_ids']);
        }
    }


    /**
     * match whether the reindexing should be fired
     * @param Mage_Index_Model_Event $event
     * @return bool
     */
    public function matchEvent(Mage_Index_Model_Event $event)
    {
        $data = $event->getNewData();
        if (isset($data[self::EVENT_MATCH_RESULT_KEY])) {
            return $data[self::EVENT_MATCH_RESULT_KEY];
        }
        $entity = $event->getEntity();
        $result = true;
        if($entity != Mage_Catalog_Model_Product::ENTITY){
            return;
        }
        $event->addNewData(self::EVENT_MATCH_RESULT_KEY, $result);
        return $result;
    }

    /**
     * Rebuild all index data
     */
    public function reindexAll()
    {
    $this->doReindexAll();
    }

Now we will explain all the methods and variables, so you will be able to modify them according to the logic of your module. Here is the explanation:

getName and getDescription return the name and the description of your index in the indexes list in the admin panel.
matchEvent is the function that returns true if the event is matched to the index process and false otherwise. As described in previous section, this function is called multiple times, thus it is common to implement here some tiny caching mechanism (the if (isset($data[self::EVENT_MATCH_RESULT_KEY])) { …).
_matchedEntitiesarray is also used by Mage_Index_Model_Indexer::indexEvents method to match the event against the event entity and the event type. (Thus for instance, the following call
```
Mage::getSingleton('index/indexer')->indexEvents(
Mage_CatalogInventory_Model_Stock_Item::ENTITY, Mage_Index_Model_Event::TYPE_DELETE
);
```
will cause the match to return false, since the _matchedEntities doesn't contain Mage_CatalogInventory_Model_Stock_Item::ENTITY key. )
_registerEvent adds some necessary data to the event object, to be used in the future in the _processEvent method
_processEvent processes the event using the event object data
reindexAll – this method is running when the user triggers the reindexing on the backend. The system assumes that after run of reindexAll, all the related data to this index will be in correct state. For instance, in the price index, we would implement here a functionality that goes through all products and refreshes the related prices data, according to the catalog rules that are active in the system.

Summing up