Uncategorized

Use TIMESTAMPDIFF to identify long running cronjobs in Magento2

Improving performance in cron jobs can be very time consuming. Even just figuring out where to start can take hours.

I came up with an idea that really helped me to locate potential performance killer just by checking the execution time by using TIMESTAMPDIFF. For example:

SELECT schedule_id, job_code, executed_at, finished_at, TIMESTAMPDIFF(SECOND, executed_at, finished_at) AS runtime
FROM cron_schedule
ORDER BY runtime desc
LIMIT 10;

SELECT schedule_id, job_code, executed_at, finished_at, TIMESTAMPDIFF(SECOND, executed_at, finished_at) AS runtime

FROM cron_schedule

ORDER BY runtime desc

LIMIT 10;

Output example:

+-------------+------------------------------------+---------------------+---------------------+---------+
| schedule_id | job_code                           | executed_at         | finished_at         | runtime |
+-------------+------------------------------------+---------------------+---------------------+---------+
|    25432321 | custom_customer_sync               | 2022-09-13 18:50:07 | 2022-09-13 18:57:28 |     456 |
|    25433099 | indexer_update_all_views           | 2022-09-13 19:05:07 | 2022-09-13 19:05:13 |       6 |
|    25432315 | indexer_update_all_views           | 2022-09-13 18:35:07 | 2022-09-13 18:35:11 |       4 |
|    25432645 | indexer_update_all_views           | 2022-09-13 18:43:06 | 2022-09-13 18:43:09 |       3 |
|    25433048 | indexer_update_all_views           | 2022-09-13 18:59:06 | 2022-09-13 18:59:09 |       3 |
...
+-------------+------------------------------------+---------------------+---------------------+---------+
20 rows in set (0.01 sec)

+-------------+------------------------------------+---------------------+---------------------+---------+

+-------------+------------------------------------+---------------------+---------------------+---------+

| 25432321 | custom_customer_sync | 2022-09-13 18:50:07 | 2022-09-13 18:57:28 | 456 |

| 25433099 | indexer_update_all_views | 2022-09-13 19:05:07 | 2022-09-13 19:05:13 | 6 |

| 25432315 | indexer_update_all_views | 2022-09-13 18:35:07 | 2022-09-13 18:35:11 | 4 |

| 25432645 | indexer_update_all_views | 2022-09-13 18:43:06 | 2022-09-13 18:43:09 | 3 |

| 25433048 | indexer_update_all_views | 2022-09-13 18:59:06 | 2022-09-13 18:59:09 | 3 |

...

+-------------+------------------------------------+---------------------+---------------------+---------+

20 rows in set (0.01 sec)

The runtime can be a sign of high memory or CPU usage, therefore, you should probably look at those cron jobs first.

Warning: array_replace_recursive(): in Setup/Declaration/Schema/Diff/Diff.php

Today I’ve noticed an error during a test deployment that occurred right after setup:upgrade. Unfortunately it didn’t indicate the affected module.

Updating modules:
Cache cleared successfully
Schema creation/updates:
Warning: array_replace_recursive(): Expected parameter 2 to be an array, null given in /var/www/html/vendor/magento/framework/Setup/Declaration/Schema/Diff/Diff.php on line 146

Updating modules:

Cache cleared successfully

Schema creation/updates:

Warning: array_replace_recursive(): Expected parameter 2 to be an array, null given in /var/www/html/vendor/magento/framework/Setup/Declaration/Schema/Diff/Diff.php on line 146

I took a quick look at the recent changes and noticed a syntax error in db_schema_whitelist.json file. In this particular case it was a comma that shouldn’t be there.

Easy fix!

Before:

"constraint": {
    "PRIMARY": true,
}

"constraint": {

"PRIMARY": true,

}

After:

"constraint": {
    "PRIMARY": true
}

"constraint": {

"PRIMARY": true

}

After removing the comma, the deployment ran through gain.

Download Magento 1 Extensions after M1 EOL

Magento 1 extensions were removed from https://marketplace.magento.com about a month ago. According to Adobe’s recent blog posts, all extensions will be removed early August from their repository, which causes a lot of frustration on developers who are still in the middle of moving towards Magento 2 or simply need the code to translate it into Magento 2 modules.

July 7, 2020 – Removal of all M1 extension listings from marketplace.magento.com. Extension will be available on repo.magento.com until 8/6/2020.
August 6, 2020 – Removal of M1 extensions from repo.magento.com

1 2	July 7, 2020 – Removal of all M1 extension listings from marketplace.magento.com. Extension will be available on repo.magento.com until 8/6/2020. August 6, 2020 – Removal of M1 extensions from repo.magento.com

Source: https://community.magento.com/t5/Magento-DevBlog/How-Extension-Developers-Can-Prepare-for-M1-End-of-Life/ba-p/446216

We encourage Magento 1 merchants to download the Magento 1 extensions they plan to keep, since Magento 1 extensions will not be available in the Magento Marketplace after July 7, 2020, and will be removed from the Magento repository after August 6, 2020.

1	We encourage Magento 1 merchants to download the Magento 1 extensions they plan to keep, since Magento 1 extensions will not be available in the Magento Marketplace after July 7, 2020, and will be removed from the Magento repository after August 6, 2020.

Source: https://magento.com/blog/magento-news/support-magento-1-software-ends-june-30-2020

So, in order to download an M1 package, you simply have to point to a specific URL. Luckily they didn’t turn off https://connect20.magentocommerce.com yet, but I believe Magento will get rid of it very soon.

https://connect20.magentocommerce.com/community/%extKey%/%version%/%extKey%-%version%.tgz

1	https://connect20.magentocommerce.com/community/%extKey%/%version%/%extKey%-%version%.tgz

I’ve spent a few minutes writing a ” quick and dirty ” PHP script that downloads the most recent version of each extension in https://connect20.magentocommerce.com/community/packages.xml including package details such as name, description, and author. For example https://connect20.magentocommerce.com/community/BankPayment/1.2.0/package.xml

<?xml version="1.0"?>
<package>
    <name>BankPayment</name>
    <version>1.2.0</version>
    <stability>stable</stability>
    <license uri="http://opensource.org/licenses/osl-3.0.php">OSL</license>
    <channel>community</channel>
    <extends/>
    <summary>Vorkasse - Bank Prepayment module which notifies the buyer to which bank account he has to transfer the money.</summary>
    <description>Bank Prepayment module which notifies the buyer to which bank account he has to transfer the money.</description>
    <notes>IBAN ready&#xD;
Allow partial capturing&#xD;
Localization improvements</notes>
    <authors><author><name>Phoenix Media GmbH</name><user>PHOENIX</user><email>bjoern.kraus@phoenix-media.eu</email></author></authors>
    <date>2017-09-05</date>
    <time>11:08:47</time>
    <contents><target name="magecommunity"><dir name="Phoenix"><dir name="BankPayment"><dir name="Block"><dir name="Adminhtml"><file name="BankAccount.php" hash="80df5881081ded22b182ac2bde3b0a3e"/>$
    <compatible/>
    <dependencies><required><php><min>5.2.0</min><max>7.0.22</max></php><extension><name>Core</name><min/><max/></extension></required></dependencies>
</package>

<?xml version="1.0"?>

<name>BankPayment</name>

<stability>stable</stability>

<channel>community</channel>

<summary>Vorkasse - Bank Prepayment module which notifies the buyer to which bank account he has to transfer the money.</summary>

<description>Bank Prepayment module which notifies the buyer to which bank account he has to transfer the money.</description>

<notes>IBAN ready

Allow partial capturing

Localization improvements</notes>

<authors><author><name>Phoenix Media GmbH</name><user>PHOENIX</user><email>bjoern.kraus@phoenix-media.eu</email></author></authors>

<contents><target name="magecommunity"><dir name="Phoenix"><dir name="BankPayment"><dir name="Block"><dir name="Adminhtml"><file name="BankAccount.php" hash="80df5881081ded22b182ac2bde3b0a3e"/>$

</package>

If you download all packages, you will end up with about 1300 extensions.

├── zoe_myzoe
│   ├── Zoe_Myzoe--package.xml
│   └── Zoe_Myzoe-.tgz
├── zoe_myzoe-1.0.0
│   ├── Zoe_Myzoe-1.0.0--package.xml
│   └── Zoe_Myzoe-1.0.0-.tgz
├── zopim_live_chat
│   ├── Zopim_Live_Chat-0.5.4-package.xml
│   └── Zopim_Live_Chat-0.5.4.tgz
└── zubibu
    ├── zubibu-1.0.0-package.xml
    └── zubibu-1.0.0.tgz

1359 directories, 2716 files

├── zoe_myzoe

│ ├── Zoe_Myzoe--package.xml

│ └── Zoe_Myzoe-.tgz

├── zoe_myzoe-1.0.0

│ ├── Zoe_Myzoe-1.0.0--package.xml

│ └── Zoe_Myzoe-1.0.0-.tgz

├── zopim_live_chat

│ ├── Zopim_Live_Chat-0.5.4-package.xml

│ └── Zopim_Live_Chat-0.5.4.tgz

└── zubibu

├── zubibu-1.0.0-package.xml

└── zubibu-1.0.0.tgz

1359 directories, 2716 files

However, it seems the packages.xml file contains a lot of extensions that are actually not available anymore, that leads to a 404.

PHP Warning:  file_get_contents(https://connect20.magentocommerce.com/community/SUMOHeavy_Coupons/1.1.0/SUMOHeavy_Coupons-1.1.0.tgz): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found
 in /var/www/extensions/index.php on line 41
PHP Warning:  file_get_contents(https://connect20.magentocommerce.com/community/SUMOHeavy_Coupons/1.1.0/package.xml): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found
 in /var/www/extensions/index.php on line 50

PHP Warning: file_get_contents(https://connect20.magentocommerce.com/community/SUMOHeavy_Coupons/1.1.0/SUMOHeavy_Coupons-1.1.0.tgz): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found

in /var/www/extensions/index.php on line 41

PHP Warning: file_get_contents(https://connect20.magentocommerce.com/community/SUMOHeavy_Coupons/1.1.0/package.xml): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found

in /var/www/extensions/index.php on line 50

Another good resource is https://packages.firegento.com or http://freegento.com/ddl-magento-extension.php to download M1 extensions.

<?php
/**
 * Connect20
 *
 * @category    Connect20
 * @author      Tobias Forkel (http://tobias.forkel.me)
 * @license     http://opensource.org/licenses/osl-3.0.php Open Software License (OSL 3.0)
 */

$packagesUrl = 'https://connect20.magentocommerce.com/community/packages.xml';
$releasesUrl = 'https://connect20.magentocommerce.com/community/%extKey%/releases.xml';
$packageUrl = 'https://connect20.magentocommerce.com/community/%extKey%/%version%/package.xml';
$downloadUrl = 'https://connect20.magentocommerce.com/community/%extKey%/%version%/%extKey%-%version%.tgz';
$packages = implode('/', [dirname(__FILE__), 'packages.xml']);

// Cache packages.xml file
if (!file_exists($packages)) {
    file_put_contents($packages, file_get_contents($packagesUrl));
}

$packages = simplexml_load_file($packages);

foreach ($packages as $data) {

    $extKey = (string)$data->n;
    $stable = (string)$data->r->s;
    $beta = (string)$data->r->b;

    $version = (!empty($stable)) ? $stable : $beta;

    if (empty($version)) {

        echo sprintf('Skipped ( %s )', $extKey);
        echo PHP_EOL;
        
        continue;
    }

    $folder = implode('/', [dirname(__FILE__), 'data', strtolower($extKey)]);

    // Create extension folder
    if (!file_exists($folder)) {
        mkdir($folder, 0777, true);
    }

    $download = str_replace(['%extKey%', '%version%'] ,[$extKey, $version], $downloadUrl);
    $downloadPath = implode('/', [$folder, basename($download)]);

    // Skip downloaded files
    if (file_exists($downloadPath)) {
        continue;
    }

    // Download the *.tgz file ( stable or beta )
    if (file_put_contents($downloadPath, file_get_contents($download))) {

        echo $download;
        echo PHP_EOL;
    }

    // Download content ( e.g description, author etc. ) from package.xml
    $package = str_replace(['%extKey%', '%version%'] ,[$extKey, $version], $packageUrl);

    if (file_put_contents(implode('/', [$folder, $extKey . '-' . $version . '-' . basename($package)]), file_get_contents($package))) {

        echo $package;
        echo PHP_EOL;
    }
}

?>

<?php

/**

* Connect20

* @category Connect20

* @author Tobias Forkel (http://tobias.forkel.me)

* @license http://opensource.org/licenses/osl-3.0.php Open Software License (OSL 3.0)

$packagesUrl = 'https://connect20.magentocommerce.com/community/packages.xml';

$releasesUrl = 'https://connect20.magentocommerce.com/community/%extKey%/releases.xml';

$packageUrl = 'https://connect20.magentocommerce.com/community/%extKey%/%version%/package.xml';

$downloadUrl = 'https://connect20.magentocommerce.com/community/%extKey%/%version%/%extKey%-%version%.tgz';

$packages = implode('/', [dirname(__FILE__), 'packages.xml']);

// Cache packages.xml file

if (!file_exists($packages)) {

file_put_contents($packages, file_get_contents($packagesUrl));

}

$packages = simplexml_load_file($packages);

foreach ($packages as $data) {

$extKey = (string)$data->n;

$stable = (string)$data->r->s;

$beta = (string)$data->r->b;

$version = (!empty($stable)) ? $stable : $beta;

if (empty($version)) {

echo sprintf('Skipped ( %s )', $extKey);

echo PHP_EOL;

continue;

}

$folder = implode('/', [dirname(__FILE__), 'data', strtolower($extKey)]);

// Create extension folder

if (!file_exists($folder)) {

mkdir($folder, 0777, true);

}

$download = str_replace(['%extKey%', '%version%'] ,[$extKey, $version], $downloadUrl);

$downloadPath = implode('/', [$folder, basename($download)]);

// Skip downloaded files

if (file_exists($downloadPath)) {

continue;

}

// Download the *.tgz file ( stable or beta )

if (file_put_contents($downloadPath, file_get_contents($download))) {

echo $download;

echo PHP_EOL;

}

// Download content ( e.g description, author etc. ) from package.xml

$package = str_replace(['%extKey%', '%version%'] ,[$extKey, $version], $packageUrl);

if (file_put_contents(implode('/', [$folder, $extKey . '-' . $version . '-' . basename($package)]), file_get_contents($package))) {

echo $package;

echo PHP_EOL;

}

Feel free to change or share the code.

Useful Postfix Commands

If your server is configured to send emails with postfix, you should be familiar with some basic Postfix commands without any help from your sysadmin.

1. List your default and custom settings

If you have added your custom Postfix settings or set your own email aliases with virtual_alias_maps, you will find it with postconf -n.

$ postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
broken_sasl_auth_clients = yes
....
virtual_alias_maps = hash:/etc/postfix/virtual

$ postconf -n

alias_database = hash:/etc/aliases

alias_maps = hash:/etc/aliases

broken_sasl_auth_clients = yes

....

virtual_alias_maps = hash:/etc/postfix/virtual

If you want to see your Postfix default settings, just change the parameter -n to -d.

$ postconf -d
connection_cache_status_update_time = 600s
connection_cache_ttl_limit = 2s
...
virtual_mailbox_domains = $virtual_mailbox_maps
virtual_mailbox_limit = 51200000

$ postconf -d

connection_cache_status_update_time = 600s

connection_cache_ttl_limit = 2s

...

virtual_mailbox_domains = $virtual_mailbox_maps

virtual_mailbox_limit = 51200000

It’s a good practise to keep this output as a running configuration snapshot in a simple text file in case you have any troubles with Postfix after server reboot or software updates.

$ postconf -d > ~/YYYYMMDD.postfix.default
$ postconf -n > ~/YYYYMMDD.postfix.custom

1 2	$ postconf -d > ~/YYYYMMDD.postfix.default $ postconf -n > ~/YYYYMMDD.postfix.custom

2. List the current queue and email id’s

Check your postqueue frequently to make sure all your emails were delivered correctly. If everything is okay you will see the message “Mail queue is empty”. Otherwise, you will see a list of undelivered emails.

$ postqueue -p
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
1B1CDC3312     2102 Tue Feb 17 00:00:01  root@web.yourserver.org
(conversation with smtp.yourserver.org[72.2.43.23] timed out while receiving the initial server greeting)
noreply@otherserver.com

94DD1C031A     1217 Fri Feb 20 00:00:02  root@web.yourserver.org
(conversation with smtp.yourserver.org[72.2.43.23] timed out while receiving the initial server greeting)
noreply@otherserver.com

$ postqueue -p

-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------

1B1CDC3312 2102 Tue Feb 17 00:00:01 root@web.yourserver.org

(conversation with smtp.yourserver.org[72.2.43.23] timed out while receiving the initial server greeting)

noreply@otherserver.com

94DD1C031A 1217 Fri Feb 20 00:00:02 root@web.yourserver.org

(conversation with smtp.yourserver.org[72.2.43.23] timed out while receiving the initial server greeting)

noreply@otherserver.com

3. Read emails from the queue

$ postcat -q 94DD1C031A

1	$ postcat -q 94DD1C031A

4. Delete emails from the queue

$ postsuper -d 94DD1C031A

1	$ postsuper -d 94DD1C031A