Posts Tagged ‘mysql’

A tool to synchronise Magento database between servers

Monday, June 28th, 2010

我时不时需要在测试服务器上加载生产服务器的实时数据,以前都是把数据下载到本地的测试服务器后,手工键入一些命令完成数据加载,每次都要花费几分钟时间。为了避免一再“浪费”这几分钟,我今天一次性投入了几小时完成了一个 php 脚本。虽然这是为 magento 的数据迁移而写的脚本,但我写完一看,用在其他地方也是可以的。

为了安全起见,该脚本是用 php 命令行运行的,所有输出针对 terminal 美化,不是 browser。保存源码为 data_mover.php,同一目录下要有 mysqldump 得到的经 gzip 的 sql 文件,文件名以 FILENAME_PREFIX 开头,以 .sql.gz 结尾。启动时只需键入

/path/to/php -f data_mover.php

即可。

初始化 PDO 对象时,按理只需要 host=localhost,不需要 unix_socket=MYSQL_SOCKET。但奇怪的是,如果通过 apache 调用本程序(虽然不是本程序的初衷,但我希望它在浏览器下也能运行),仅指定 host=localhost 作 PDO __construct() 参数,会产生一个莫名其妙的错误:

SQLSTATE[HY000] [2002] Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock’ (2)

似乎是 PDO bug。可以通过 host=127.0.0.1 或者追加 unix_socket=MYSQL_SOCKET 来避免。如果通过 php 命令行启动则没有这个问题。

而后,还涉及怎么删除所有数据表的问题。看似简单的一个问题,我发现 mysql 竟然没有一个类似于 DROP/TRUNCATE TABLE * 单行命令。于是除了本脚本用的方法外,我还想了不下两种办法:

其一,删掉整个数据库重新创建。
mysqladmin -f -h localhost -u (DB_ROOT_USERNAME) -p(DB_ROOT_PASSWORD) drop (DB_NAME)
mysqladmin -h localhost -u (DB_ROOT_USERNAME) -p(DB_ROOT_PASSWORD) create (DB_NAME)
但这需要比数据表操作更高权限的用户,在这个无关大局的脚本里去使用高权限的用户的密码,实在非我所愿。

其二,是我 google 来的,方法很巧,但很可惜,因为 foreign keys 的存在,运行这条命令会出错。
mysqldump -u (DB_USERNAME) -p(DB_PASSWORD) –add-drop-table –no-data (DB_NAME) | grep ^DROP | mysql -u (DB_USERNAME) -p(DB_PASSWORD) (DB_NAME)


<?php

define('MYSQL_SOCKET', '/path/to/mysql/socket');
define('DB_NAME', 'db_name');
define('DB_USERNAME', 'db_username');
define('DB_PASSWORD', 'db_password');
define('FILENAME_PREFIX', 'filename_prefix');
define('TEST_URL', 'http://test.domain/');

if ($handle = opendir(dirname(__FILE__))) {
$found = false;
/* This is the correct way to loop over the directory. */
while (false !== ($file = readdir($handle))) {
if (substr($file, 0, 12) == FILENAME_PREFIX && substr($file, -7) == '.sql.gz') {
if ($found) {
//compare which one is newer
if (filemtime($file) > filemtime($fileFound)) {
$fileFound = $file;
}
}
else {
$found = true;
$fileFound = $file;
}
}
}

if ($found) {
echo "Found the newest file $fileFound, and will work on it.\n";
try {
$pdo = new PDO(
'mysql:host=localhost;dbname=' . DB_NAME . ';unix_socket=' . MYSQL_SOCKET,
DB_USERNAME,
DB_PASSWORD,
array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8")
);
echo "Connected to database.\n";
}
catch (Exception $e) {
die ("Error occurred when connecting to database. Quoting error message: " . $e->getMessage() . "\n");
}
$sql = 'SET FOREIGN_KEY_CHECKS = 0';
$pdo->prepare($sql)->execute();

/* query all tables */
$sql = "SHOW TABLES FROM ". DB_NAME;
$stmt = $pdo->prepare($sql);
$stmt->execute();
$result = $stmtDale->fetchAll(PDO::FETCH_COLUMN, 0);
$magentoTableNames = array();
/* add table name to array */
foreach ($result as $tableName) {
if (substr($tableName, 0, 8) == 'magento_') {
$magentoTableNames[] = $tableName;
}
}

//drop all magento tables
$count = count($magentoTableNames);
if ($count > 0) {
$sql = 'DROP TABLE '. implode(',', $magentoTableNames);
$pdo->prepare($sql)->execute();
echo "Found and dropped $count magento tables.\n";
}
else {
echo "No existing magento tables found.\n";
}

//import data via pipe
echo "Importing data. It may take a while...\n";
$output = shell_exec("gunzip < $fileFound | mysql -h localhost -u " . DB_USERNAME . " -p" . DB_PASSWORD . " " . DB_NAME);
echo "Importing data completed.\n";
if ($output) {
echo "This is output during data import:\n$output\n";
}

//after import, change some data for test domains
$sql = 'UPDATE magento_core_config_data SET value=? WHERE scope=? AND scope_id=? AND path=?';
$stmt = $pdo->prepare($sql);
$stmt->execute(array(TEST_URL, 'default', 0, 'web/unsecure/base_url'));
$stmt->execute(array(TEST_URL, 'default', 0, 'web/secure/base_url'));
$stmt->execute(array(TEST_URL, 'websites', 2, 'web/unsecure/base_url'));
$stmt->execute(array(TEST_URL, 'websites', 2, 'web/secure/base_url'));
echo "Config data changed to fit testing environment.\n";
echo "All missions completed.\n";
}
else {
echo "File not found. What else can I do?\n";
}
closedir($handle);
}
else {
echo "File access not permitted.\n";
}

How magento deducts stock level

Tuesday, March 16th, 2010

我心里一直有个疑问,收到订单后,Magento 是怎么更新库存数量的?是直接用一个新库存数字去更新,还是用 upadte magento_cataloginventory_stock_item set qty=qty-qty_ordered 之类的办法。

通过跟踪 sql 语句,我感觉前者的可能性很大。

UPDATE `magento_cataloginventory_stock_item` SET `item_id` = '125', `product_id` = '125', `stock_id` = '1', `qty` = '93.000000', `min_qty` = '0.000000', `use_config_min_qty` = '1', `is_qty_decimal` = '0', `backorders` = '1', `use_config_backorders` = '0', `min_sale_qty` = '1.000000', `use_config_min_sale_qty` = '1', `max_sale_qty` = '0.000000', `use_config_max_sale_qty` = '1', `is_in_stock` = '1', `low_stock_date` = NULL, `notify_stock_qty` = '0.000000', `use_config_notify_stock_qty` = '1', `manage_stock` = '1', `use_config_manage_stock` = '0', `stock_status_changed_automatically` = '0', `shelf` = '' WHERE (item_id='125')

这样很不好,如果几个人同时买同一个产品,会搞乱库存数。不知道被破坏的几率有多大,有空要搞一个并发下单的测试。

Mysql varchar could be transformed to memo in Ms Access

Thursday, February 11th, 2010

Mysql varchar could be transformed to memo in Ms Access when Mysql table is used as linked table inside Access, but I want to prevent this from happening. Mainly because I can not join tables with memo field.

As I observed, long Mysql varchar field is transformed to memo field, but short one stays as text field. The maximum length of varchar is 85 which allows varchar stays as text field in Access.

This limitation is quite touch, isn’t it?

Magento database fetch mode

Wednesday, October 14th, 2009

I was used to write $modelName->column_name to get attribute value in database. It requires $resource->_getReadAdapter()->setFetchMode(Zend_Db::FETCH_OBJ);.

However, it is not Magento’s default fetch mode (default is Zend_Db:FETCH_ASSOC). Whenever fetch mode is changed, it must be changed back to the default after using it. Otherwise, method getCollection() of ALL models will stop working. There may be other side effects if you leave the adapter’s fetch mode changed.

mysql_update

Tuesday, August 11th, 2009

When using phpmyadmin, after select a database, then click on Privileges, you might run into a error saying

Unknown column ‘Event_priv’ in ‘field list’

The error will go after you run

mysql_upgrade

in command line.

I feel mysql query speed is improved after running “mysql_update”.

Speed bottleneck of the web server

Tuesday, August 4th, 2009

非常典型的多层架构:
第一层 Nginx
第二层 php fastcgi
第三层 memcached
第四层 MySql

Apache 有个 mod_php,相当于合并了第一层和第二层,Nginx 没有 module for php,这不是什么问题,分层更利于扩展。第三层的加入完全是为了减轻数据库压力,提高性能。目前第2,3,4层之间的优化差不多到极限了(或者说到我能力的极限了),但第1,2层之间尚有潜力可挖。

not_in_use.php 和 not_in_use.html 都是一个静态文件,没有数据库操作。但 php 文件必须由 Nginx 经由 php fastcgi (使用 unix socket)产生,html 则由 Nginx 直接访问文件系统,就单因素分析,php fastcgi 是普通文件系统速度的34%,所以要想办法绕开 php fastcgi。以下测试在数据中心主机上直接运行 ApacheBench。

测试一:
$ ab -kc 100 -n 500 http://magento/not_in_use.php
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking magento (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Finished 500 requests

Server Software: nginx/0.6.36
Server Hostname: magento
Server Port: 80

Document Path: /not_in_use.php
Document Length: 7686 bytes

Concurrency Level: 100
Time taken for tests: 0.336355 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 4089329 bytes
HTML transferred: 4004406 bytes
Requests per second: 1486.52 [#/sec] (mean)
Time per request: 67.271 [ms] (mean)
Time per request: 0.673 [ms] (mean, across all concurrent requests)
Transfer rate: 11871.39 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 11 17.8 3 58
Processing: 13 49 17.9 50 94
Waiting: 3 45 18.8 46 88
Total: 25 60 14.2 61 94

Percentage of the requests served within a certain time (ms)
50% 61
66% 70
75% 72
80% 74
90% 79
95% 80
98% 84
99% 86
100% 94 (longest request)

=======================================================
测试二:
$ ab -kc 100 -n 500 http://magento/not_in_use.html
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking magento (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Finished 500 requests

Server Software: nginx/0.6.36
Server Hostname: magento
Server Port: 80

Document Path: /not_in_use.html
Document Length: 7686 bytes

Concurrency Level: 100
Time taken for tests: 0.115725 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Keep-Alive requests: 500
Total transferred: 3959000 bytes
HTML transferred: 3843000 bytes
Requests per second: 4320.59 [#/sec] (mean)
Time per request: 23.145 [ms] (mean)
Time per request: 0.231 [ms] (mean, across all concurrent requests)
Transfer rate: 33406.78 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 3 7.3 0 22
Processing: 7 17 4.1 18 23
Waiting: 7 16 4.0 17 23
Total: 7 20 9.9 18 41

Percentage of the requests served within a certain time (ms)
50% 18
66% 20
75% 22
80% 37
90% 39
95% 40
98% 41
99% 41
100% 41 (longest request)

Zend_Db ignore default charset of Mysql

Sunday, April 26th, 2009

While I set my Mysql database default character set as UTF-8, Zend_Db_Adapter does not pick up the setting. Zend_Db write to Mysql using its own default character set (I guess it’s ISO-8859-1), unless it is specifically told by

$dbAdapter->query(”SET NAMES ‘utf8′”);

before

Zend_Db_Table::setDefaultAdapter($dbAdapter);

Quick access to data for both internet and intranet users

Friday, January 23rd, 2009

我曾思考过,也不止一次被人问过:数据中心的数据如何可以被公司员工快速存取?Internet users(客户、供应商、合作伙伴)通常存取单条数据,数据中心与internet users之间的传输速度足够满意;但intranet users(员工)经常存取批量数据,一般的宽带会让intranet users在存取数据时有大量的时滞。

难道在intranet建数据中心?小公司是负担不起的,最起码ADSL就得撤换成SDSL甚至更快的专线,最低档的SDSL(2M)也比ADSL贵十倍。在专业的数据中心可以以很低的成本实现100M到桌面,如果要不牺牲internet user experience,买条100M的专线,那是天价。

在成本不上升的情况下,如何让internet和intranet用户同时能快速访问数据?我曾想过Ajax,但觉得Ajax不是好的方案,因为Ajax只能缓解intranet users在等待数据时的焦虑心理,数据传输速度没有得到改善;我还想过分解数据表,把internet users经常用到的数据放在数据中心,把intranet suers经常用到的数据放在本地,但总有一些数据是双方都经常用到的,很难权衡该放在哪一端;而且对于一个现成的系统,分解数据表似乎不太现实。

今天我很高兴,因为我发现MySQL早就内置了数据同步功能(只是我孤陋寡闻罢了),只要几步简单设置,一个复杂的internet / intranet 矛盾就解决了,幸好我还没有走入Ajax和分解数据表的歧路。

做事一定要站得高看得远啊,这也是我说“多找软件、用好软件”的初衷。

Workaround to row cannot be located for updating

Tuesday, January 13th, 2009

用ADO + ODBC操作Mysql database,用起来总不顺手。经常有莫名其妙的错误,其中不乏connector bugs。这些bugs不是我等能解决的,只有尽量不去触发bugs才是上策。

比如,我今天在使用recordset.update时被提示”Row cannot be located for updating. Some values may have been changed since it was last read.” 我排错了好久,最后总结出以下best practices:

  • 避免使用”SELECT *”来查询,用到哪些字段就SELECT那些字段;
  • 如果SELECT中的字段中,某些值没有改变,recordset.update 可能会出错(也就是说,所有字段值都被改变后才能update)
  • Connector/ODBC Application Specific Tips特别提到Access can’t always handle the MySQL DATE column properly. If you have a problem with these, change the columns to DATETIME.
  • Float或double类型字段值,避免用Float或Double运算后的值直接去更新,应该保留必要精度后的值去更新。例如,x和y都是double类型,total字段也是double类型,不要用x+y去更新total字段,若total只要保留两位小数,就用formatnumber(x+y, 2)去更新total。

综上,为了简单起见,如果要update a record,干脆先删除这条record,再recordset.addnew,赋予新值。对于新鲜记录的更新,以上四种可能的错误均没有出现。