Bugfixing

Use TIMESTAMPDIFF to identify long running cronjobs in Magento2

Improving performance in cron jobs can be very time consuming. Even just figuring out where to start can take hours.

I came up with an idea that really helped me to locate potential performance killer just by checking the execution time by using TIMESTAMPDIFF. For example:

SELECT schedule_id, job_code, executed_at, finished_at, TIMESTAMPDIFF(SECOND, executed_at, finished_at) AS runtime
FROM cron_schedule
ORDER BY runtime desc
LIMIT 10;

SELECT schedule_id, job_code, executed_at, finished_at, TIMESTAMPDIFF(SECOND, executed_at, finished_at) AS runtime

FROM cron_schedule

ORDER BY runtime desc

LIMIT 10;

Output example:

+-------------+------------------------------------+---------------------+---------------------+---------+
| schedule_id | job_code                           | executed_at         | finished_at         | runtime |
+-------------+------------------------------------+---------------------+---------------------+---------+
|    25432321 | custom_customer_sync               | 2022-09-13 18:50:07 | 2022-09-13 18:57:28 |     456 |
|    25433099 | indexer_update_all_views           | 2022-09-13 19:05:07 | 2022-09-13 19:05:13 |       6 |
|    25432315 | indexer_update_all_views           | 2022-09-13 18:35:07 | 2022-09-13 18:35:11 |       4 |
|    25432645 | indexer_update_all_views           | 2022-09-13 18:43:06 | 2022-09-13 18:43:09 |       3 |
|    25433048 | indexer_update_all_views           | 2022-09-13 18:59:06 | 2022-09-13 18:59:09 |       3 |
...
+-------------+------------------------------------+---------------------+---------------------+---------+
20 rows in set (0.01 sec)

+-------------+------------------------------------+---------------------+---------------------+---------+

+-------------+------------------------------------+---------------------+---------------------+---------+

| 25432321 | custom_customer_sync | 2022-09-13 18:50:07 | 2022-09-13 18:57:28 | 456 |

| 25433099 | indexer_update_all_views | 2022-09-13 19:05:07 | 2022-09-13 19:05:13 | 6 |

| 25432315 | indexer_update_all_views | 2022-09-13 18:35:07 | 2022-09-13 18:35:11 | 4 |

| 25432645 | indexer_update_all_views | 2022-09-13 18:43:06 | 2022-09-13 18:43:09 | 3 |

| 25433048 | indexer_update_all_views | 2022-09-13 18:59:06 | 2022-09-13 18:59:09 | 3 |

...

+-------------+------------------------------------+---------------------+---------------------+---------+

20 rows in set (0.01 sec)

The runtime can be a sign of high memory or CPU usage, therefore, you should probably look at those cron jobs first.

Apply patches with composers post-install-cmd

I just want to introduce another approach that will apply Magento patches, in case your Magento instance isn’t hosted on Adobe’s Cloud infrastructure or can’t be applied as explained in How to apply a composer patch provided by Adobe for some reason.

1. Update your composer.json file

Add a new section “scripts” at the end of your composer.json file and validate your file with composer validate.

"scripts": {
    "post-install-cmd": [
        "patch -p1 < ./patches/MDVA-43395_EE_2.4.3-p1_COMPOSER_v1.patch"
    ]
}

"scripts": {

"post-install-cmd": [

"patch -p1 < ./patches/MDVA-43395_EE_2.4.3-p1_COMPOSER_v1.patch"

]

}

If you see ./composer.json is valid you are good to move on. Btw. your composer.lock file doesn’t need to be updated.

2. Create a folder ./patches/

Create a folder ./patches/ or something similar and copy your *.patch file in there.

3. Test it

Run composer install and wait. Your script and patch should be executed after Generating autoload files just like in the below output.

Generating autoload files
98 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
> patch -p1 < ./patches/MDVA-43395_EE_2.4.3-p1_COMPOSER_v1.patch
patching file vendor/magento/module-email/Model/Template/Filter.php
patching file vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php

Generating autoload files

98 packages you are using are looking for funding.

Use the `composer fund` command to find out more!

> patch -p1 < ./patches/MDVA-43395_EE_2.4.3-p1_COMPOSER_v1.patch

patching file vendor/magento/module-email/Model/Template/Filter.php

patching file vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php

4. Deploy it

Commit your changes ( composer.json and ./patches/ ) folder and run your deployment procedure.

That’s it!

Adobe APSB22-12 – Don’t panic!

Adobe has released a critical security patch early this week on https://support.magento.com/hc/en-us/articles/4426353041293-Security-updates-available-for-Adobe-Commerce-APSB22-12– and https://helpx.adobe.com/security/products/magento/apsb22-12.html.

Of course, Adobe Commerce merchants and agencies are nervous and want to apply the security patch as soon as possible. However, I believe as long as your admin passwords are strong enough and you have control about your admin accounts, you shouldn’t be worried too much.

Here is why.

Let’s have a look at the content of the security patch.

diff --git a/vendor/magento/module-email/Model/Template/Filter.php b/vendor/magento/module-email/Model/Template/Filter.php
index 1a7c3683820a..586cb485ee1f 100644
--- a/vendor/magento/module-email/Model/Template/Filter.php
+++ b/vendor/magento/module-email/Model/Template/Filter.php
@@ -618,6 +618,12 @@ public function transDirective($construction)
         }
 
         $text = __($text, $params)->render();
+
+        $pattern = '/{{.*?}}/';
+        do {
+            $text = preg_replace($pattern, '', (string)$text);
+        } while (preg_match($pattern, $text));
+
         return $this->applyModifiers($text, $modifiers);
     }
 
diff --git a/vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php b/vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php
index f2fe398c3848..78034d70ba51 100644
--- a/vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php
+++ b/vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php
@@ -55,6 +55,11 @@ public function process(array $construction, Template $filter, array $templateVa
             $result = $this->filterApplier->applyFromRawParam($construction['filters'], $result);
         }
 
+        $pattern = '/{{.*?}}/';
+        do {
+            $result = preg_replace($pattern, '', (string)$result);
+        } while (preg_match($pattern, $result));
+
         return $result;
     }

diff --git a/vendor/magento/module-email/Model/Template/Filter.php b/vendor/magento/module-email/Model/Template/Filter.php

index 1a7c3683820a..586cb485ee1f 100644

--- a/vendor/magento/module-email/Model/Template/Filter.php

+++ b/vendor/magento/module-email/Model/Template/Filter.php

@@ -618,6 +618,12 @@ public function transDirective($construction)

}

$text = __($text, $params)->render();

+ $pattern = '/{{.*?}}/';

+ do {

+ $text = preg_replace($pattern, '', (string)$text);

+ } while (preg_match($pattern, $text));

return $this->applyModifiers($text, $modifiers);

}

diff --git a/vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php b/vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php

index f2fe398c3848..78034d70ba51 100644

--- a/vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php

+++ b/vendor/magento/framework/Filter/DirectiveProcessor/VarDirective.php

@@ -55,6 +55,11 @@ public function process(array $construction, Template $filter, array $templateVa

$result = $this->filterApplier->applyFromRawParam($construction['filters'], $result);

}

+ $pattern = '/{{.*?}}/';

+ do {

+ $result = preg_replace($pattern, '', (string)$result);

+ } while (preg_match($pattern, $result));

return $result;

}

The patch will update two files only that are responsible to process template variables such as email variables {{var logo_url}} or {{config path=”general/store_information/name”}} or variables within CMS pages or Static Blocks. The change is trivial and will wipe out unwanted content by using the pattern $pattern = ‘/{{.*?}}/’.

It basically means, in order to be able to exploit this issue, you must:

1. Know the admin url of a store
2. Have access to the Magento backend ( ideally admin privileges )
3. Know how to exploit the issue

So, as long as you maintain admin accounts regularly and know who is actively working in your backend, you should be fine.

However, if you have modules installed that add custom variables by injecting Magento\Email\Model\Template\Filter in combination with 3rd party services, you should probably go and install the patch with your next release.

Update: 2022/02/16

Code could possibly injected by using the customers billing or shipping address ( e.g {{trans “%name,” name=$order.getBillingAddress().getName()}} ) depending on how strong form validation is in the front-end ( My Account > Addresses or Checkout ).

Magento Cloud – sendmail: /etc/msmtprc: line 6: command host needs an argument

This morning I ran a deployment on a fresh Magento Cloud environment and suddenly got the below SMTP error right after setup:upgrade, which was very unusual.

[2021-02-23 06:42:08] INFO: Running setup upgrade.
        [2021-02-23 06:56:37] INFO: Set flag: var/.deploy_is_failed
        [2021-02-23 06:56:37] ERROR: [126] The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade --keep-generated --ansi --no-interaction  | tee -a /app/var/log/install_upgrade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an argument
        [2021-02-23 06:56:37] INFO: Set flag: var/.deploy_is_failed
        W: 
        W: In Processor.php line 129:
        W:                                                                                
        W:   The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade  
        W:    --keep-generated --ansi --no-interaction  | tee -a /app/var/log/install_up  
        W:   grade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an a  
        W:   rgument                                                                      
        W:                                                                                
        W: 
        W: In InstallUpdate.php line 100:
        W:                                                                                
        W:   The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade  
        W:    --keep-generated --ansi --no-interaction  | tee -a /app/var/log/install_up  
        W:   grade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an a  
        W:   rgument                                                                      
        W:                                                                                
        W: 
        W: In Setup.php line 58:
        W:                                                                                
        W:   The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade  
        W:    --keep-generated --ansi --no-interaction  | tee -a /app/var/log/install_up  
        W:   grade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an a  
        W:   rgument                                                                      
        W:                                                                                
        W: 
        W: In Shell.php line 86:
        W:                                                                                
        W:   The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade  
        W:    --keep-generated --ansi --no-interaction  | tee -a /app/var/log/install_up  
        W:   grade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an a  
        W:   rgument                                                                      
        W:                                                                                
        W: 
        W: deploy
        W:

[2021-02-23 06:42:08] INFO: Running setup upgrade.

[2021-02-23 06:56:37] INFO: Set flag: var/.deploy_is_failed

[2021-02-23 06:56:37] ERROR: [126] The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade --keep-generated --ansi --no-interaction | tee -a /app/var/log/install_upgrade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an argument

[2021-02-23 06:56:37] INFO: Set flag: var/.deploy_is_failed

W: In Processor.php line 129:

W: The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade

W: --keep-generated --ansi --no-interaction | tee -a /app/var/log/install_up

W: grade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an a

W: rgument

W: In InstallUpdate.php line 100:

W: The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade

W: --keep-generated --ansi --no-interaction | tee -a /app/var/log/install_up

W: grade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an a

W: rgument

W: In Setup.php line 58:

W: The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade

W: --keep-generated --ansi --no-interaction | tee -a /app/var/log/install_up

W: grade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an a

W: rgument

W: In Shell.php line 86:

W: The command "/bin/bash -c "set -o pipefail; php ./bin/magento setup:upgrade

W: --keep-generated --ansi --no-interaction | tee -a /app/var/log/install_up

W: grade.log"" failed. sendmail: /etc/msmtprc: line 6: command host needs an a

W: rgument

W: deploy

I had a look at the recent changes and found a mail() function in the modules InstallSchema.php script.

$to_email = 'sdfsdf@sdfsdf.com';
$subject = 'New Installation of sdfsdfsdf M2 extension';
$headers = "MIME-Version: 1.0" . "\r\n";
$headers .= "Content-type:text/html;charset=UTF-8" . "\r\n";
$headers .= 'From: ' . $merchantmail . '' . "\r\n";
mail($to_email, $subject, $body, $headers);

$to_email = 'sdfsdf@sdfsdf.com';

$subject = 'New Installation of sdfsdfsdf M2 extension';

$headers = "MIME-Version: 1.0" . "\r\n";

$headers .= "Content-type:text/html;charset=UTF-8" . "\r\n";

$headers .= 'From: ' . $merchantmail . '' . "\r\n";

mail($to_email, $subject, $body, $headers);

In this particular case, a 3rd party module developer has implemented a tracking system to notify his agency about new module installations. It may work on some hosting platforms, but obviously not on Magento Cloud during the build process.

October CMS – Widget OC Media Manager

If you work with the form elements fileupload or mediafinder in combination with user admin roles ( Settings > Administrators ) you may end up with the following error while saving a model.

"A widget with class name 'ocmediamanager' has not been bound to the controller" on line 597 of /var/www/src/modules/backend/classes/Controller.php

1	"A widget with class name 'ocmediamanager' has not been bound to the controller" on line 597 of /var/www/src/modules/backend/classes/Controller.php

In order to fix that, you must review the permissions of your admin user account for Upload and manage media contents – images, videos, sounds, documents. and select Allow or Inherit, depending on how you setup your roles.

After saving the permissions, you should be able to upload or select an image and save the model.

October CMS – How To Fix Broken Thumbnails

I recently started setting up a dockerized October CMS environment based on Alpine Linux, MariaDB and PHP-FPM which was actually a very straightforward process. However, for some reason the integrated image resizer didn’t create thumbnails of uploaded images.

Unfortunately there was nothing in the log files which could help me to identify the cause of the problem. After checking the file permissions, I decided to trace back the issue and found the reason in file vendor/october/rain/src/Database/Attach/Resizer.php.

The method getMimeType() returned always null, which broke the entire image manipulation process.

// Get the file extension
$this->extension = $file->guessExtension();
$this->mime = $file->getMimeType();

// Get the file extension

$this->extension = $file->guessExtension();

$this->mime = $file->getMimeType();

After a quick research I found out that getMimeType requires php_fileinfo which I simply forgot to set in my Dockerfile. After adding php7-fileinfo and rebuilding the container, the issue was finally fixed.

Notice: Use of undefined constant T_CURLY_OPEN – assumed ‘T_CURLY_OPEN’ in /var/www/src/setup/src/ Magento/Setup/Module/ Di/Code/Reader/FileClassScanner.php on line 72

Today I finally had some time to cleaned up my Dockerfile for Magento 2. I am using Alpine Linux for most of my Magento 2 projects which usually includes NGINX, PHP-FPM and MariaDB.

# Use Alpine Linux
FROM alpine:3.7

# Declare maintainer
MAINTAINER Tobias Forkel <tobias@forkel.me>

# Timezone
ENV TIMEZONE Australia/Melbourne

RUN addgroup -g 1000 www && \
    adduser -S www -G www -s /sbin/nologin -D

ADD https://repos.php.earth/alpine/phpearth.rsa.pub /etc/apk/keys/phpearth.rsa.pub
RUN echo "https://repos.php.earth/alpine/v3.7" >> /etc/apk/repositories

RUN apk --no-cache add \
    bash \
    git \
	openssl \
	nginx \
	supervisor \
	curl \
	mariadb \
	mariadb-client \
	libxml2-utils \
	php7.1-common \
	php7.1-fpm \
	php7.1-json \
....

# Use Alpine Linux

FROM alpine:3.7

# Declare maintainer

MAINTAINER Tobias Forkel <tobias@forkel.me>

# Timezone

ENV TIMEZONE Australia/Melbourne

RUN addgroup -g 1000 www && \

adduser -S www -G www -s /sbin/nologin -D

ADD https://repos.php.earth/alpine/phpearth.rsa.pub /etc/apk/keys/phpearth.rsa.pub

RUN echo "https://repos.php.earth/alpine/v3.7" >> /etc/apk/repositories

RUN apk --no-cache add \

bash \

git \

openssl \

nginx \

supervisor \

curl \

mariadb \

mariadb-client \

libxml2-utils \

php7.1-common \

php7.1-fpm \

php7.1-json \

....

While testing my updated Dockerfile I came across the following PHP notice which prevented the deployment scripts from finishing the Magento 2 setup.

Notice: Use of undefined constant T_CURLY_OPEN - assumed 'T_CURLY_OPEN' in /var/www/src/setup/src/Magento/Setup/Module/ Di/Code/Reader/FileClassScanner.php on line 72

1	Notice: Use of undefined constant T_CURLY_OPEN - assumed 'T_CURLY_OPEN' in /var/www/src/setup/src/Magento/Setup/Module/ Di/Code/Reader/FileClassScanner.php on line 72

The problem was simply a missing PHP extension php7.1-tokenizer which has resolved the issue after re-creating the container with –build.

docker-compose up -d --build

1	docker-compose up -d --build

PHP message: PHP Fatal error: Interface ‘SessionHandlerInterface’ not found in /var/www/src/vendor/ laravel/framework/src/Illuminate/Session/FileSessionHandler.php on line 10

Today I successfully installed my first October CMS project locally via composer / command line. The installation was pretty straightforward and worked surprisingly well on my Alpine Linux docker container which includes NGINX, PHP-FPM and MariaDB.

October.Demo
- v1.0.1:  First version of Demo
Seeded System 
Seeded Backend 
.=========================================.
                ,@@@@@@@,                  
        ,,,.   ,@@@@@@/@@,  .oo8888o.      
     ,&%%&%&&%,@@@@@/@@@@@@,8888\88/8o     
    ,%&\%&&%&&%,@@@\@@@/@@@88\88888/88'    
    %&&%&%&/%&&%@@\@@/ /@@@88888\88888'    
    %&&%/ %&%%&&@@\ V /@@' `88\8 `/88'     
    `&%\ ` /%&'    |.|        \ '|8'       
        |o|        | |         | |         
        |.|        | |         | |         
`========= INSTALLATION COMPLETE ========='

October.Demo

- v1.0.1: First version of Demo

Seeded System

Seeded Backend

.=========================================.

,@@@@@@@,

,,,. ,@@@@@@/@@, .oo8888o.

,&%%&%&&%,@@@@@/@@@@@@,8888\88/8o

,%&\%&&%&&%,@@@\@@@/@@@88\88888/88'

%&&%&%&/%&&%@@\@@/ /@@@88888\88888'

%&&%/ %&%%&&@@\ V /@@' `88\8 `/88'

`&%\ ` /%&' |.| \ '|8'

|o| | | | |

|.| | | | |

`========= INSTALLATION COMPLETE ========='

However, when I tried to open the front-end I got the following error.

PHP message: PHP Fatal error:  Interface 'SessionHandlerInterface' not found in /var/www/src/vendor/laravel/framework/src/Illuminate/Session/FileSessionHandler.php on line 10

1	PHP message: PHP Fatal error: Interface 'SessionHandlerInterface' not found in /var/www/src/vendor/laravel/framework/src/Illuminate/Session/FileSessionHandler.php on line 10

After a quick research I found the solution on Stackoverflow. I basically forgot to install the PHP extension php7-session inside my docker container. I manually installed the missing extension, but also updated my Dockerfile in case I have to rebuild my container.

bash-4.4# apk add php7-session
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/1) Installing php7-session (7.1.33-r0)
OK: 291 MiB in 87 packages

bash-4.4# apk add php7-session

fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz

fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz

(1/1) Installing php7-session (7.1.33-r0)

OK: 291 MiB in 87 packages

After that I manually killed all PHP processes to make sure php-fpm loads the new extension with the next page reload.

pkill php

pkill php

UI Components and searchResultToOutput in Magento2

Today I had to fix the following issue in a custom module. The error appeared on the UI Component list view.

Fatal error: Method Magento\Ui\TemplateEngine\Xhtml\Result::__toString() must not throw an exception, caught TypeError: Argument 1 passed to Magento\Framework\View\Element\UiComponent\DataProvider\DataProvider::searchResultToOutput() must implement interface Magento\Framework\Api\Search\SearchResultInterface, instance of [Namespace]\[Module]\Model\ResourceModel\Team\Collection given, called in

Fatal error: Method Magento\Ui\TemplateEngine\Xhtml\Result::__toString() must not throw an exception, caught TypeError: Argument 1 passed to Magento\Framework\View\Element\UiComponent\DataProvider\DataProvider::searchResultToOutput() must implement interface Magento\Framework\Api\Search\SearchResultInterface, instance of [Namespace]\[Module]\Model\ResourceModel\Team\Collection given, called in

I found the reason of the problem in the di.xml file. All what was missing was the following virtualType node. After adding this and re-compiling the code, the list view was fixed.

<virtualType name="[Namespace]\[Module]\Model\ResourceModel\Team\Collection" type="Magento\Framework\View\Element\UiComponent\DataProvider\SearchResult">
    <arguments>
        <argument name="mainTable" xsi:type="string">[namespace]_[module]_team</argument>
        <argument name="resourceModel" xsi:type="string">[Namespace]\[Module]\Model\ResourceModel\Team</argument>
    </arguments>
</virtualType>

<argument name="mainTable" xsi:type="string">[namespace]_[module]_team</argument>

<argument name="resourceModel" xsi:type="string">[Namespace]\[Module]\Model\ResourceModel\Team</argument>

</arguments>

</virtualType>

UI Component Listing And Not registered handle in Magento2

Today I created a new listing for a custom module, but unfortunately all what I got was the error message below.

Not registered handle [namespace]_[module]_team_index_data_source

1	Not registered handle [namespace]_[module]_team_index_data_source

After digging around for few minutes I noticed the above handler was actually missing in the di.xml file. I simply added the following collections argument and re-compiled the code.

<type name="Magento\Framework\View\Element\UiComponent\DataProvider\CollectionFactory">
    <arguments>
        <argument name="collections" xsi:type="array">
            ...
            <item name="[namespace]_[module]_team_index_data_source" xsi:type="string">[Namespace]\[Module]\Model\ResourceModel\Team\Collection</item>
            ...
        </argument>
    </arguments>
</type>

...

<item name="[namespace]_[module]_team_index_data_source" xsi:type="string">[Namespace]\[Module]\Model\ResourceModel\Team\Collection</item>

...

</argument>

</arguments>

</type>

All good!