Linux VPS流量查看/监测工具 -- vnStat

vnStat是一个应用于Linux或BSD平台，对网卡流量做监控。 因为并非是基于网络包嗅探的方式而是基于/proc的分析。现在vnStat已经有出vnstat PHP frontend 扩展了，可以以PHP脚本的形式直接调用vnStat监测的数据，如下图：

vnstat1

vnstat2

下面说一下具体的安装步骤：

下载：
wget http://soft.vpser.net/status/vnstat/vnstat-1.10.tar.gz
解压：
tar xvzf vnstat-1.10.tar.gz
进入目录：
cd vnstat-1.10/
编译安装：
make
make install
如果64位平台上编译的话用make 64bit命令

这样vnstat 就安装好了。下面讲vnstat的工作运行配置。

下载最新版vnstat PHP frontend，

地址：http://soft.vpser.net/status/vnstat/vnstat_php_frontend-1.5.1.tar.gz

解压文件，改名为vnstat，然后上传到某个网站根目录下，这样就可以以http://www.vpser.net/vnstat 来访问到，就出来页面了，但此时没有数据，因为还要使系统生成数据。

建立流量数据库:
ifconfig 指令查看需要监控的网卡,假设只有eth0
然后生成数据库:
/usr/bin/vnstat -u -i eth0
然后定时更新数据库,通过cron的方式：

实际上在安装vnstat时，已经在系统里安装好了vnstat的cron，crontab内容在/etc/cron.d/vnstat，如果没有使用命令touch /etc/cron.d/vnstat 创建。
内容为:
0-55/5 *        * * *   root   vnstat -u -i eth0
0-55/5 *        * * *   root   vnstat --dumpdb -i eth0 >/var/lib/vnstat/vnstat_dump_eth0

第一行为安装时自动指定的.

第二行是为了更新eth0的数据后，dump出来一个文件，给 php访问接口访问.

这里dump出来的vnstat_dump_eth0 文件名是有规定的。

# 注意：有时候采用上面的收集办法，无法采集到数据，可以采用下面的办法

cat > /var/lib/vnstat/dump.sh<<EOF
vnstat -u -i eth0
vnstat --dumpdb -i eth0 >/var/lib/vnstat/vnstat_dump_eth0
EOF

再执行：
crontab -e

加入执行命令

*/5 * * * * sh /var/lib/vnstat/dump.sh

再删除vnstat原有的cron

rm /etc/cron.d/vnstat

在vnstat_php_frontend-1.5.1.tar.gz 包里的config.php 里有说明。

最后，编辑　vnstat/config.php 这个文件，将数据目录改为　/var/lib/vnstat/，如果不存在此目录，需要使用mkdir -p /var/lib/vnstat/ 创建此目录。
即：
$data_dir = '/var/lib/vnstat/';

指定显示标题

$iface_title['eth0'] = 'VPSer-Linode';

下面设置以什么图片格式显示，可以是svg或png，一般选择png就可以，svg在IE下需要安装插件，Firefox不需要；如果用png在IE、Firefox下均能直接显示。

$graph_format='png';

指定vnstat的安装目录，如果是安装上述方法安装，直接按下面填写即可。

$vnstat_bin = '/usr/bin/vnstat';

将要监控的网卡指定为１个：

// list of network interfaces monitored by vnStat
//$iface_list = array(’eth0′, ‘eth1′, ’sixxs’);
$iface_list = array(’eth0′,);

// will be displayed instead
//
$iface_title['eth0'] = ‘Internal’;
//$iface_title['eth1'] = ‘Internet’;
//$iface_title['sixxs'] = ‘SixXS IPv6′;

修改语言为英语：$language = 'nl'; 将nl替换为en 保存。

 

------------------------------------------------------

Linux查看网络即时网速

sar -n DEV 1 100

1代表一秒统计并显示一次
100代表统计一百次

crontab命令

前一天学习了 at 命令是针对仅运行一次的任务，循环运行的例行性计划任务，linux系统则是由 cron (crond) 这个系统服务来控制的。Linux 系统上面原本就有非常多的计划性工作，因此这个系统服务是默认启动的。另外, 由于使用者自己也可以设置计划任务，所以， Linux 系统也提供了使用者控制计划任务的命令 :crontab 命令。

一、crond简介

crond是linux下用来周期性的执行某种任务或等待处理某些事件的一个守护进程，与windows下的计划任务类似，当安装完成操作系统后，默认会安装此服务工具，并且会自动启动crond进程，crond进程每分钟会定期检查是否有要执行的任务，如果有要执行的任务，则自动执行该任务。

Linux下的任务调度分为两类，系统任务调度和用户任务调度。

系统任务调度：系统周期性所要执行的工作，比如写缓存数据到硬盘、日志清理等。在/etc目录下有一个crontab文件，这个就是系统任务调度的配置文件。

/etc/crontab文件包括下面几行：

[root@localhost ~]# cat /etc/crontab

SHELL=/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin

MAILTO=""HOME=/

# run-parts

51 * * * * root run-parts /etc/cron.hourly

24 7 * * * root run-parts /etc/cron.daily

22 4 * * 0 root run-parts /etc/cron.weekly

42 4 1 * * root run-parts /etc/cron.monthly

[root@localhost ~]#

前四行是用来配置crond任务运行的环境变量，第一行SHELL变量指定了系统要使用哪个shell，这里是bash，第二行PATH变量指定了系统执行命令的路径，第三行MAILTO变量指定了crond的任务执行信息将通过电子邮件发送给root用户，如果MAILTO变量的值为空，则表示不发送任务执行信息给用户，第四行的HOME变量指定了在执行命令或者脚本时使用的主目录。第六至九行表示的含义将在下个小节详细讲述。这里不在多说。

用户任务调度：用户定期要执行的工作，比如用户数据备份、定时邮件提醒等。用户可以使用 crontab 工具来定制自己的计划任务。所有用户定义的crontab 文件都被保存在 /var/spool/cron目录中。其文件名与用户名一致。

使用者权限文件：

文件：

/etc/cron.deny

说明：

该文件中所列用户不允许使用crontab命令

文件：

/etc/cron.allow

说明：

该文件中所列用户允许使用crontab命令

文件：

/var/spool/cron/

说明：

所有用户crontab文件存放的目录,以用户名命名

crontab文件的含义：

用户所建立的crontab文件中，每一行都代表一项任务，每行的每个字段代表一项设置，它的格式共分为六个字段，前五段是时间设定段，第六段是要执行的命令段，格式如下：

minute   hour   day   month   week   command

其中：

minute： 表示分钟，可以是从0到59之间的任何整数。

hour：表示小时，可以是从0到23之间的任何整数。

day：表示日期，可以是从1到31之间的任何整数。

month：表示月份，可以是从1到12之间的任何整数。

week：表示星期几，可以是从0到7之间的任何整数，这里的0或7代表星期日。

command：要执行的命令，可以是系统命令，也可以是自己编写的脚本文件。

在以上各个字段中，还可以使用以下特殊字符：

星号（*）：代表所有可能的值，例如month字段如果是星号，则表示在满足其它字段的制约条件后每月都执行该命令操作。

逗号（,）：可以用逗号隔开的值指定一个列表范围，例如，“1,2,5,7,8,9”

中杠（-）：可以用整数之间的中杠表示一个整数范围，例如“2-6”表示“2,3,4,5,6”

正斜线（/）：可以用正斜线指定时间的间隔频率，例如“0-23/2”表示每两小时执行一次。同时正斜线可以和星号一起使用，例如*/10，如果用在minute字段，表示每十分钟执行一次。

二、crond服务

安装crontab：

yum install crontabs

服务操作说明：

/sbin/service crond start //启动服务

/sbin/service crond stop //关闭服务

/sbin/service crond restart //重启服务

/sbin/service crond reload //重新载入配置

查看crontab服务状态：

service crond status

手动启动crontab服务：

service crond start

查看crontab服务是否已设置为开机启动，执行命令：

ntsysv

加入开机自动启动：

chkconfig –level 35 crond on

三、crontab命令详解

1．命令格式：

crontab [-u user] file

crontab [-u user] [ -e | -l | -r ]

2．命令功能：

通过crontab 命令，我们可以在固定的间隔时间执行指定的系统指令或 shell script脚本。时间间隔的单位可以是分钟、小时、日、月、周及以上的任意组合。这个命令非常设合周期性的日志分析或数据备份等工作。

3．命令参数：

-u user：用来设定某个用户的crontab服务，例如，“-u ixdba”表示设定ixdba用户的crontab服务，此参数一般有root用户来运行。

file：file是命令文件的名字,表示将file做为crontab的任务列表文件并载入crontab。如果在命令行中没有指定这个文件，crontab命令将接受标准输入（键盘）上键入的命令，并将它们载入crontab。

-e：编辑某个用户的crontab文件内容。如果不指定用户，则表示编辑当前用户的crontab文件。

-l：显示某个用户的crontab文件内容，如果不指定用户，则表示显示当前用户的crontab文件内容。

-r：从/var/spool/cron目录中删除某个用户的crontab文件，如果不指定用户，则默认删除当前用户的crontab文件。

-i：在删除用户的crontab文件时给确认提示。

4．常用方法：

1). 创建一个新的crontab文件

在考虑向cron进程提交一个crontab文件之前，首先要做的一件事情就是设置环境变量EDITOR。cron进程根据它来确定使用哪个编辑器编辑crontab文件。9 9 %的UNIX和LINUX用户都使用vi，如果你也是这样，那么你就编辑$ HOME目录下的. profile文件，在其中加入这样一行：

EDITOR=vi; export EDITOR

然后保存并退出。不妨创建一个名为<user> cron的文件，其中<user>是用户名，例如， davecron。在该文件中加入如下的内容。

# (put your own initials here)echo the date to the console every

# 15minutes between 6pm and 6am

0,15,30,45 18-06 * * * /bin/echo 'date' > /dev/console

保存并退出。确信前面5个域用空格分隔。

在上面的例子中，系统将每隔1 5分钟向控制台输出一次当前时间。如果系统崩溃或挂起，从最后所显示的时间就可以一眼看出系统是什么时间停止工作的。在有些系统中，用tty1来表示控制台，可以根据实际情况对上面的例子进行相应的修改。为了提交你刚刚创建的crontab文件，可以把这个新创建的文件作为cron命令的参数：

$ crontab davecron

现在该文件已经提交给cron进程，它将每隔1 5分钟运行一次。

同时，新创建文件的一个副本已经被放在/var/spool/cron目录中，文件名就是用户名(即dave)。

2). 列出crontab文件

为了列出crontab文件，可以用：

$ crontab -l

0,15,30,45,18-06 * * * /bin/echo date > dev/tty1

你将会看到和上面类似的内容。可以使用这种方法在$ H O M E目录中对crontab文件做一备份：

$ crontab -l > $HOME/mycron

这样，一旦不小心误删了crontab文件，可以用上一节所讲述的方法迅速恢复。

3). 编辑crontab文件

如果希望添加、删除或编辑crontab文件中的条目，而E D I TO R环境变量又设置为v i，那么就可以用v i来编辑crontab文件，相应的命令为：

$ crontab -e

可以像使用v i编辑其他任何文件那样修改crontab文件并退出。如果修改了某些条目或添加了新的条目，那么在保存该文件时， c r o n会对其进行必要的完整性检查。如果其中的某个域出现了超出允许范围的值，它会提示你。

我们在编辑crontab文件时，没准会加入新的条目。例如，加入下面的一条：

# DT:delete core files,at 3.30am on 1,7,14,21,26,26 days of each month

30 3 1,7,14,21,26 * * /bin/find -name "core' -exec rm {} ;

现在保存并退出。最好在crontab文件的每一个条目之上加入一条注释，这样就可以知道它的功能、运行时间，更为重要的是，知道这是哪位用户的作业。

现在让我们使用前面讲过的crontab -l命令列出它的全部信息：

$ crontab -l

# (crondave installed on Tue May 4 13:07:43 1999)

# DT:ech the date to the console every 30 minites

0,15,30,45 18-06 * * * /bin/echo date > /dev/tty1

# DT:delete core files,at 3.30am on 1,7,14,21,26,26 days of each month

30 3 1,7,14,21,26 * * /bin/find -name "core' -exec rm {} ;

4). 删除crontab文件

要删除crontab文件，可以用：

$ crontab -r

5). 恢复丢失的crontab文件

如果不小心误删了crontab文件，假设你在自己的$ H O M E目录下还有一个备份，那么可以将其拷贝到/var/spool/cron/<username>，其中<username>是用户名。如果由于权限问题无法完成拷贝，可以用：

$ crontab <filename>

其中，<filename>是你在$ H O M E目录中副本的文件名。

我建议你在自己的$ H O M E目录中保存一个该文件的副本。我就有过类似的经历，有数次误删了crontab文件（因为r键紧挨在e键的右边）。这就是为什么有些系统文档建议不要直接编辑crontab文件，而是编辑该文件的一个副本，然后重新提交新的文件。

有些crontab的变体有些怪异，所以在使用crontab命令时要格外小心。如果遗漏了任何选项，crontab可能会打开一个空文件，或者看起来像是个空文件。这时敲delete键退出，不要按<Ctrl-D>，否则你将丢失crontab文件。

5．使用实例

实例1：每1分钟执行一次command

命令：

* * * * * command

 

实例2：每小时的第3和第15分钟执行

命令：

3,15 * * * * command

 

实例3：在上午8点到11点的第3和第15分钟执行

命令：

3,15 8-11 * * * command

 

实例4：每隔两天的上午8点到11点的第3和第15分钟执行

命令：

3,15 8-11 */2 * * command

 

实例5：每个星期一的上午8点到11点的第3和第15分钟执行

命令：

3,15 8-11 * * 1 command

 

实例6：每晚的21:30重启smb

命令：

30 21 * * * /etc/init.d/smb restart

 

实例7：每月1、10、22日的4 : 45重启smb

命令：

45 4 1,10,22 * * /etc/init.d/smb restart

 

实例8：每周六、周日的1 : 10重启smb

命令：

10 1 * * 6,0 /etc/init.d/smb restart

 

实例9：每天18 : 00至23 : 00之间每隔30分钟重启smb

命令：

0,30 18-23 * * * /etc/init.d/smb restart

 

实例10：每星期六的晚上11 : 00 pm重启smb

命令：

0 23 * * 6 /etc/init.d/smb restart

 

实例11：每一小时重启smb

命令：

* */1 * * * /etc/init.d/smb restart

 

实例12：晚上11点到早上7点之间，每隔一小时重启smb

命令：

* 23-7/1 * * * /etc/init.d/smb restart

 

实例13：每月的4号与每周一到周三的11点重启smb

命令：

0 11 4 * mon-wed /etc/init.d/smb restart

 

实例14：一月一号的4点重启smb

命令：

0 4 1 jan * /etc/init.d/smb restart

实例15：每小时执行/etc/cron.hourly目录内的脚本

命令：

01   *   *   *   *     root run-parts /etc/cron.hourly

说明：

run-parts这个参数了，如果去掉这个参数的话，后面就可以写要运行的某个脚本名，而不是目录名了

四、使用注意事项

1. 注意环境变量问题

有时我们创建了一个crontab，但是这个任务却无法自动执行，而手动执行这个任务却没有问题，这种情况一般是由于在crontab文件中没有配置环境变量引起的。

在crontab文件中定义多个调度任务时，需要特别注意的一个问题就是环境变量的设置，因为我们手动执行某个任务时，是在当前shell环境下进行的，程序当然能找到环境变量，而系统自动执行任务调度时，是不会加载任何环境变量的，因此，就需要在crontab文件中指定任务运行所需的所有环境变量，这样，系统执行任务调度时就没有问题了。

不要假定cron知道所需要的特殊环境，它其实并不知道。所以你要保证在shelll脚本中提供所有必要的路径和环境变量，除了一些自动设置的全局变量。所以注意如下3点：

1）脚本中涉及文件路径时写全局路径；

2）脚本执行要用到java或其他环境变量时，通过source命令引入环境变量，如：

cat start_cbp.sh

#!/bin/sh

source /etc/profile

export RUN_CONF=/home/d139/conf/platform/cbp/cbp_jboss.conf

/usr/local/jboss-4.0.5/bin/run.sh -c mev &

3）当手动执行脚本OK，但是crontab死活不执行时。这时必须大胆怀疑是环境变量惹的祸，并可以尝试在crontab中直接引入环境变量解决问题。如：

0 * * * * . /etc/profile;/bin/sh /var/www/java/audit_no_count/bin/restart_audit.sh

2. 注意清理系统用户的邮件日志

每条任务调度执行完毕，系统都会将任务输出信息通过电子邮件的形式发送给当前系统用户，这样日积月累，日志信息会非常大，可能会影响系统的正常运行，因此，将每条任务进行重定向处理非常重要。

例如，可以在crontab文件中设置如下形式，忽略日志输出：

0 */3 * * * /usr/local/apache2/apachectl restart >/dev/null 2>&1

“/dev/null 2>&1”表示先将标准输出重定向到/dev/null，然后将标准错误重定向到标准输出，由于标准输出已经重定向到了/dev/null，因此标准错误也会重定向到/dev/null，这样日志输出问题就解决了。

3. 系统级任务调度与用户级任务调度

系统级任务调度主要完成系统的一些维护操作，用户级任务调度主要完成用户自定义的一些任务，可以将用户级任务调度放到系统级任务调度来完成（不建议这么做），但是反过来却不行，root用户的任务调度操作可以通过“crontab –uroot –e”来设置，也可以将调度任务直接写入/etc/crontab文件，需要注意的是，如果要定义一个定时重启系统的任务，就必须将任务放到/etc/crontab文件，即使在root用户下创建一个定时重启系统的任务也是无效的。

4. 其他注意事项

新创建的cron job，不会马上执行，至少要过2分钟才执行。如果重启cron则马上执行。

当crontab突然失效时，可以尝试/etc/init.d/crond restart解决问题。或者查看日志看某个job有没有执行/报错tail -f /var/log/cron。

千万别乱运行crontab -r。它从Crontab目录（/var/spool/cron）中删除用户的Crontab文件。删除了该用户的所有crontab都没了。

在crontab中%是有特殊含义的，表示换行的意思。如果要用的话必须进行转义%，如经常用的date ‘+%Y%m%d’在crontab里是不会执行的，应该换成date ‘+%Y%m%d’。

Description of Exif file format

What is Exif file format?

Basically, Exif file format is the same as JPEG file format. Exif inserts some of image/digicam information data and thumbnail image to JPEG in conformity to JPEG specification. Therefore you can view Exif format image files by JPEG compliant Internet browser/Picture viewer/Photo retouch software etc. as a usual JPEG image files.

JPEG format and Marker

Every JPEG file starts from binary value '0xFFD8', ends by binary value '0xFFD9'. There are several binary 0xFFXX data in JPEG data, they are called as "Marker", and it means the period of JPEG information data. 0xFFD8 means SOI(Start of image), 0xFFD9 means EOI(End of image). These two special Markers have no data following, the other Markers have data with it. Basic format of Marker is below.

0xFF+Marker Number(1 byte)+Data size(2 bytes)+Data(n bytes)

Data size(2 Bytes) has "Motorola" byte align, starts from bigger digits. Please notice that "Data" contains Data size descriptor, if there is a Marker like this;

FF C1 00 0C

It means this Marker(0xFFC1) has 0x000C(equal 12)bytes of data. But the data size '12' includes "Data size" descriptor, it follows only 10 bytes of data after 0x000C.

In JPEG format, some of Markers describe data, then SOS(Start of stream) Marker placed. After the SOS marker, JPEG image stream starts and terminated by EOI Marker.

SOI Marker Marker XX size=SSSS Marker YY size=TTTT SOS Marker size=UUUU Image stream EOI Marker
FFD8 FFXX SSSS DDDD...... FFYY TTTT DDDD...... FFDA UUUU DDDD.... I I I I.... FFD9

 

Marker used by Exif

The marker 0xFFE0~0xFFEF is named "Application Marker", not necessary for decoding JPEG image. They are used by user application. For example, older olympus/canon/casio/agfa digicams use JFIF(JPEG File Interchange Format) for storing images. JFIF uses APP0(0xFFE0) Marker for inserting digicam configuration data and thumbnail image.

Also Exif uses an Application Marker for inserting data, but Exif uses APP1(0xFFE1) Marker to avoid a conflict with JFIF format. Every Exif file formats starts from this format;

SOI Marker APP1 Marker APP1 Data Other Marker
FFD8 FFE1 SSSS 457869660000 TTTT...... FFXX SSSS DDDD......

It starts from SOI(0xFFD8) Marker, so it's a JPEG file. Then APP1 Marker follows immediately. All the data of Exif are stored in this APP1 data area. The part of "SSSS" on upper table means the size of APP1 data area (Exif data area). Please notice that the size "SSSS" includes the size of descriptor itself also.

After the "SSSS", APP1 data starts. The first part is a special data to identify whether Exif or not, ASCII character "Exif" and 2bytes of 0x00 are used.

After the APP1 Marker area, the other JPEG Markers follows.

Exif data structure

Roughly structure of Exif data (APP1) is shown as below. This is a case of "Intel" byte align and it contains JPEG format thumbnail. As described above, Exif data is starts from ASCII character "Exif" and 2bytes of 0x00, then Exif data follows. Exif uses TIFF format to store data. For more datails of TIFF format, please refer to "TIFF6.0 specification".

FFE1 APP1 Marker
SSSS APP1 Data APP1 Data Size
45786966 0000 Exif Header
49492A00 08000000 TIFF Header
XXXX. . . . IFD0 (main image) Directory
LLLLLLLL Link to IFD1
XXXX. . . . Data area of IFD0
XXXX. . . . Exif SubIFD Directory
00000000 End of Link
XXXX. . . . Data area of Exif SubIFD
XXXX. . . . IFD1(thumbnail image) Directory
00000000 End of Link
XXXX. . . . Data area of IFD1
FFD8XXXX. . . XXXXFFD9 Thumbnail image

Structure of TIFF header

First 8bytes of TIFF format are TIFF header. First 2bytes defines byte align of TIFF data. If it is 0x4949="I I", it means "Intel" type byte align. If it is 0x4d4d="MM", it means "Motorola" type byte align. For example, value '305,419,896' is noted as 0x12345678 by sixteenth system. At the Motrola align, it is stored as 0x12,0x34,0x56,0x78. If it's Intel align, it is stored as 0x78,0x56,0x34,0x12. It seems that most of digicams uses Intel align. Ricoh uses Motorola align. Sony uses Intel align except D700. Kodak DC200/210/240 use Motorola align, but DC220/260 use Intel align though they are using PowerPC! Therefore when we need the value of Exif data, we MUST check byte align every time. Though JPEG data uses Motorola align only, Exif allows both alignment. I can't understand why Exif didn't fix a byte align to Motorola.

Next 2bytes are always 2bytes-length value of 0x002A. If the data uses Intel align, next 2bytes are "0x2a00". If it uses Motorola, they are "0x002a". The last 4bytes of TIFF header are an offset to the first IFD(Image File Directory, described in next chapter). Includes this offset, all the offset value used in TIFF format counts offset bytes from the first of TIFF header("I I" or "MM"). Usually the first IFD starts immediately next to TIFF header, so this offset has value '0x00000008'.

Byte align TAG Mark Offset to first IFD
"I I" or "MM" 2A00 0x00000008

IFD : Image file directory

Next to TIFF header, there is the first IFD:Image File Directory. It contains image information data. At the chart below, the first 2bytes('EEEE') means the number of directory entry contains in this IFD. Then directory entry (12bytes per entry) follows. After last directory entry, there is a 4bytes of data('LLLLLLLL' at the chart), it means an offset to next IFD. If its value is '0x00000000', it means this is the last IFD and there is no linked IFD.

EEEE No. of directory entry
TTTT ffff NNNNNNNN DDDDDDDD Entry 0
TTTT ffff NNNNNNNN DDDDDDDD Entry 1
. . . . . . . . . . . . . . .
TTTT ffff NNNNNNNN DDDDDDDD Entry EEEE-1
LLLLLLLL Offset to next IFD

'TTTT'(2bytes) of above chart is Tag number, this shows a kind of data. 'ffff'(2bytes) is data format, 'NNNNNNNN'(4bytes) is number of components. 'DDDDDDDD'(4bytes) contains a data value or offset to data value.

Data format

Data format ('ffff' at the above chart) is defined as below. "rational" means a fractional value, it contains 2 signed/unsigned long integer value, and the first represents the numerator, the second, the denominator.

Value 1 2 3 4 5 6
Format unsigned byte ascii strings unsigned short unsigned long unsigned rational signed byte
Bytes/component 1 1 2 4 8 1
Value 7 8 9 10 11 12
Format undefined signed short signed long signed rational single float double float
Bytes/component 1 2 4 8 4 8

You can get the total data byte length by multiplies a 'bytes/components' value (see above chart) by number of components stored 'NNNNNNNN' area. If total data length is less than 4bytes, 'DDDDDDDD' contains the value of that Tag. If its size is over 4bytes, 'DDDDDDDD' contains the offset to data stored address.

IFD data structure

At Exif format, the first IFD is IFD0(IFD of main image), then it links to IFD1(IFD of thumbnail image) and IFD link is terminated. But IFD0/IFD1 doesn't contain any digicam's information such as shutter speed, focal length etc. IFD0 always contains special Tag Exif Offset(0x8769), it shows an offset to Exif SubIFD. Exif SubIFD is IFD formatted data also, it contains digicam's information.

If the first part of TIFF data is above, it can read as;

  • The first 2bytes are "I I", byte align is 'Intel'.
  • Address 0x0004~0x0007 is 0x08000000, IFD0 starts from address '0x0008'
  • Address 0x0008~0x0009 is 0x0200, number of directory entry of IFD0 is '2'.
  • Address 0x000a~0x000b is 0x1A01, it means this is a XResolution(0x011A) Tag, it contains a horizontal resolution of image.
  • Address 0x000c~0x000d is 0x0500, format of this value is unsigned rational(0x0005).
  • Address 0x000e~0x0011 is 0x01000000, number of components is '1'. Unsigned rational's data size is 8bytes/components, so total data length is 1x8=8bytes.
  • Total data length is larger than 4bytes, so next 4bytes contains an offset to data.
  • Address 0x0012~0x0015 is 0x26000000, XResolution data is stored to address 0x0026
  • Address 0x0026~0x0029 is 0x48000000, numerator is 72, address 0x002a~0x002d is 0x0100000000, denominator is '1'. So the value of XResoultion is 72/1.
  • Address0x0016~0x0017 is 0x6987, next Tag is ExifOffset(0x8769). Its value is an offset to Exif SubIFD
  • Data format is 0x0004, unsigned long integer.
  • This Tag has one component. Unsigned long integer's data size is 4bytes/components, so total data size is 4bytes.
  • Total data size is equal to 4bytes, next 4bytes contains the value of Exif SubIFD offset.
  • Address 0x001e~0x0021 is 0x11020000, Exif SubIFD starts from address '0x0211'.
  • This is the last directory entry, next 4bytes shows an offset to next IFD.
  • Address 0x0022~0x0025 is 0x40000000, next IFD starts from address '0x0040'

 

Thumbnail image

Exif format contains thumbnail of image (except Ricoh RDC-300Z). Usually it is located next to the IFD1. There are 3 formats for thumbnails; JPEG format(JPEG uses YCbCr), RGB TIFF format, YCbCr TIFF format.

JPEG format thumbnail

If the value of Compression(0x0103) Tag in IFD1 is '6', thumbnail image format is JPEG. Most of Exif image uses JPEG format for thumbnail. In that case, you can get offset of thumbnail by JpegIFOffset(0x0201) Tag in IFD1, size of thumbnail by JpegIFByteCount(0x0202) Tag. Data format is ordinary JPEG format, starts from 0xFFD8 and ends by 0xFFD9. It seems that JPEG format and 160x120pixels of size are recommended thumbnail format for Exif2.1 or later.

TIFF format thumbnail

If the value of Compression(0x0103) Tag in IFD1 is '1', thumbnail image format is no compression(called TIFF image). Start point of thumbnail data is StripOffset(0x0111) Tag, size of thumbnail is the sum of StripByteCounts(0x0117) Tag.

If thumbnail uses no compression and PhotometricInterpretation(0x0106)Tag in IFD1 has a value '2', thumbnail uses RGB format. In that case, you can see thumbnail image by simply copy data to computer's RGB format(such as BMP format, or copy to VRAM directory). Kodak DC-210/220/260 use this format.
If that tag has a value '6', thumbnail uses YCbCr format. If you want to see thumbnail, you must convert it to RGB. Ricoh RDC4200/4300, Fuji DS-7/300 and DX-5/7/9 use this format(newer RDC5000/MX-X00 series use JPEG). Next section is brief description to conversion of Fuji DS's thumbnail. For more details, refer to TIFF6.0 specification.

At DX-5/7/9, YCbCrSubsampling(0x0212) has values of '2,1', PlanarConfiguration(0x011c) has a value '1'. So the data align of this image is below.

Y(0,0),Y(1,0),Cb(0,0),Cr(0,0), Y(2,0),Y(3,0),Cb(2,0),Cr(3.0), Y(4,0),Y(5,0),Cb(4,0),Cr(4,0). . . .

The numerics in parenthesis are pixel coordinates. DX series' YCbCrCoefficients(0x0211) has values '0.299/0.587/0.114', ReferenceBlackWhite(0x0214) has values '0,255,128,255,128,255'. Therefore to convert from Y/Cb/Cr to RGB is;

B(0,0)=(Cb-128)*(2-0.114*2)+Y(0,0)
R(0,0)=(Cr-128)*(2-0.299*2)+Y(0,0)
G(0,0)=(Y(0,0)-0.114*B(0,0)-0.299*R(0,0))/0.587

Horizontal subsampling is a value '2', so you can calculate B(1,0)/R(1,0)/G(1,0) by using the Y(1,0) and Cr(0,0)/Cb(0,0). Repeat this conversion by value of ImageWidth(0x0100) and ImageLength(0x0101).

Tag number used by Exif/TIFF

Tag numbers used by Exif/TIFF are shown as below. If the Tag has upper limit of components number, CompoNo column has numeric value. If it has no value, there is no limitation.

Tags used by IFD0 (main image)
Tag No. Tag Name Format CompoNo Desc.
0x010e ImageDescription ascii string Describes image
0x010f Make ascii string Shows manufacturer of digicam
0x0110 Model ascii string Shows model number of digicam
0x0112 Orientation unsigned short 1 The orientation of the camera relative to the scene, when the image was captured. The start point of stored data is, '1' means upper left, '3' lower right, '6' upper right, '8' lower left, '9' undefined.
0x011a XResolution unsigned rational 1 Display/Print resolution of image. Large number of digicam uses 1/72inch, but it has no mean because personal computer doesn't use this value to display/print out.
0x011b YResolution unsigned rational 1
0x0128 ResolutionUnit unsigned short 1 Unit of XResolution(0x011a)/YResolution(0x011b). '1' means no-unit, '2' means inch, '3' means centimeter.
0x0131 Software ascii string Shows firmware(internal software of digicam) version number.
0x0132 DateTime ascii string 20 Date/Time of image was last modified. Data format is "YYYY:MM:DD HH:MM:SS"+0x00, total 20bytes. In usual, it has the same value of DateTimeOriginal(0x9003)
0x013e WhitePoint unsigned rational 2 Defines chromaticity of white point of the image. If the image uses CIE Standard Illumination D65(known as international standard of 'daylight'), the values are '3127/10000,3290/10000'.
0x013f PrimaryChromaticities unsigned rational 6 Defines chromaticity of the primaries of the image. If the image uses CCIR Recommendation 709 primearies, values are '640/1000,330/1000,300/1000,600/1000,150/1000,0/1000'.
0x0211 YCbCrCoefficients unsigned rational 3 When image format is YCbCr, this value shows a constant to translate it to RGB format. In usual, values are '0.299/0.587/0.114'.
0x0213 YCbCrPositioning unsigned short 1 When image format is YCbCr and uses 'Subsampling'(cropping of chroma data, all the digicam do that), defines the chroma sample point of subsampling pixel array. '1' means the center of pixel array, '2' means the datum point.
0x0214 ReferenceBlackWhite unsigned rational 6 Shows reference value of black point/white point. In case of YCbCr format, first 2 show black/white of Y, next 2 are Cb, last 2 are Cr. In case of RGB format, first 2 show black/white of R, next 2 are G, last 2 are B.
0x8298 Copyright ascii string Shows copyright information
0x8769 ExifOffset unsigned long 1 Offset to Exif Sub IFD

Tags used by Exif SubIFD
Tag No. Tag Name Format CompoNo Desc.
0x829a ExposureTime unsigned rational 1 Exposure time (reciprocal of shutter speed). Unit is second.
0x829d FNumber unsigned rational 1 The actual F-number(F-stop) of lens when the image was taken.
0x8822 ExposureProgram unsigned short 1 Exposure program that the camera used when image was taken. '1' means manual control, '2' program normal, '3' aperture priority, '4' shutter priority, '5' program creative (slow program), '6' program action(high-speed program), '7' portrait mode, '8' landscape mode.
0x8827 ISOSpeedRatings unsigned short 2 CCD sensitivity equivalent to Ag-Hr film speedrate.
0x9000 ExifVersion undefined 4 Exif version number. Stored as 4bytes of ASCII character (like "0210")
0x9003 DateTimeOriginal ascii string 20 Date/Time of original image taken. This value should not be modified by user program.
0x9004 DateTimeDigitized ascii string 20 Date/Time of image digitized. Usually, it contains the same value of DateTimeOriginal(0x9003).
0x9101 ComponentConfiguration undefined Unknown. It seems value 0x00,0x01,0x02,0x03 always.
0x9102 CompressedBitsPerPixel unsigned rational 1 The average compression ratio of JPEG.
0x9201 ShutterSpeedValue signed rational 1 Shutter speed. To convert this value to ordinary 'Shutter Speed'; calculate this value's power of 2, then reciprocal. For example, if value is '4', shutter speed is 1/(2^4)=1/16 second.
0x9202 ApertureValue unsigned rational 1 The actual aperture value of lens when the image was taken. To convert this value to ordinary F-number(F-stop), calculate this value's power of root 2 (=1.4142). For example, if value is '5', F-number is 1.4142^5 = F5.6.
0x9203 BrightnessValue signed rational 1 Brightness of taken subject, unit is EV.
0x9204 ExposureBiasValue signed rational 1 Exposure bias value of taking picture. Unit is EV.
0x9205 MaxApertureValue unsigned rational 1 Maximum aperture value of lens. You can convert to F-number by calculating power of root 2 (same process of ApertureValue(0x9202).
0x9206 SubjectDistance signed rational 1 Distance to focus point, unit is meter.
0x9207 MeteringMode unsigned short 1 Exposure metering method. '1' means average, '2' center weighted average, '3' spot, '4' multi-spot, '5' multi-segment.
0x9208 LightSource unsigned short 1 Light source, actually this means white balance setting. '0' means auto, '1' daylight, '2' fluorescent, '3' tungsten, '10' flash.
0x9209 Flash unsigned short 1 '1' means flash was used, '0' means not used.
0x920a FocalLength unsigned rational 1 Focal length of lens used to take image. Unit is millimeter.
0x927c MakerNote undefined Maker dependent internal data. Some of maker such as Olympus/Nikon/Sanyo etc. uses IFD format for this area.
0x9286 UserComment undefined Stores user comment.
0xa000 FlashPixVersion undefined 4 Stores FlashPix version. Unknown but 4bytes of ASCII characters "0100"exists.
0xa001 ColorSpace unsigned short 1 Unknown, value is '1'.
0xa002 ExifImageWidth unsignedshort/long 1 Size of main image.
0xa003 ExifImageHeight unsignedshort/long 1
0xa004 RelatedSoundFile ascii string If this digicam can record audio data with image, shows name of audio data.
0xa005 ExifInteroperabilityOffset unsigned long 1 Extension of "ExifR98", detail is unknown. This value is offset to IFD format data. Currently there are 2 directory entries, first one is Tag0x0001, value is "R98", next is Tag0x0002, value is "0100".
0xa20e FocalPlaneXResolution unsigned rational 1 CCD's pixel density.
0xa20f FocalPlaneYResolution unsigned rational 1
0xa210 FocalPlaneResolutionUnit unsigned short 1 Unit of FocalPlaneXResoluton/FocalPlaneYResolution. '1' means no-unit, '2' inch, '3' centimeter.
0xa217 SensingMethod unsigned short 1 Shows type of image sensor unit. '2' means 1 chip color area sensor, most of all digicam use this type.
0xa300 FileSource undefined 1 Unknown but value is '3'.
0xa301 SceneType undefined 1 Unknown but value is '1'.

Tags used by IFD1 (thumbnail image)
Tag No. Tag Name Format CompoNo Desc.
0x0100 ImageWidth unsignedshort/long 1 Shows size of thumbnail image.
0x0101 ImageLength unsignedshort/long 1
0x0102 BitsPerSample unsigned short 3 When image format is no compression, this value shows the number of bits per component for each pixel. Usually this value is '8,8,8'
0x0103 Compression unsigned short 1 Shows compression method. '1' means no compression, '6' means JPEG compression.
0x0106 PhotometricInterpretation unsigned short 1 Shows the color space of the image data components. '1' means monochrome, '2' means RGB, '6' means YCbCr.
0x0111 StripOffsets unsignedshort/long When image format is no compression, this value shows offset to image data. In some case image data is striped and this value is plural.
0x0115 SamplesPerPixel unsigned short 1 When image format is no compression, this value shows the number of components stored for each pixel. At color image, this value is '3'.
0x0116 RowsPerStrip unsigned short/long 1 When image format is no compression and image has stored as strip, this value shows how many rows stored to each strip. If image has not striped, this value is the same as ImageLength(0x0101).
0x0117 StripByteConunts unsignedshort/long When image format is no compression and stored as strip, this value shows how many bytes used for each strip and this value is plural. If image has not stripped, this value is single and means whole data size of image.
0x011a XResolution unsigned rational 1 Display/Print resolution of image. Large number of digicam uses 1/72inch, but it has no mean because personal computer doesn't use this value to display/print out.
0x011b YResolution unsigned rational 1
0x011c PlanarConfiguration unsigned short 1 When image format is no compression YCbCr, this value shows byte aligns of YCbCr data. If value is '1', Y/Cb/Cr value is chunky format, contiguous for each subsampling pixel. If value is '2', Y/Cb/Cr value is separated and stored to Y plane/Cb plane/Cr plane format.
0x0128 ResolutionUnit unsigned short 1 Unit of XResolution(0x011a)/YResolution(0x011b). '1' means inch, '2' means centimeter.
0x0201 JpegIFOffset unsigned long 1 When image format is JPEG, this value show offset to JPEG data stored.
0x0202 JpegIFByteCount unsigned long 1 When image format is JPEG, this value shows data size of JPEG image.
0x0211 YCbCrCoefficients unsigned rational 3 When image format is YCbCr, this value shows constants to translate it to RGB format. In usual, '0.299/0.587/0.114' are used.
0x0212 YCbCrSubSampling unsigned short 2 When image format is YCbCr and uses subsampling(cropping of chroma data, all the digicam do that), this value shows how many chroma data subsampled. First value shows horizontal, next value shows vertical subsample rate.
0x0213 YCbCrPositioning unsigned short 1 When image format is YCbCr and uses 'Subsampling'(cropping of chroma data, all the digicam do that), this value defines the chroma sample point of subsampled pixel array. '1' means the center of pixel array, '2' means the datum point(0,0).
0x0214 ReferenceBlackWhite unsigned rational 6 Shows reference value of black point/white point. In case of YCbCr format, first 2 show black/white of Y, next 2 are Cb, last 2 are Cr. In case of RGB format, first 2 show black/white of R, next 2 are G, last 2 are B.

Misc Tags
Tag No. Tag Name Format CompoNo Desc.
0x00fe NewSubfileType unsigned long 1
0x00ff SubfileType unsigned short 1
0x012d TransferFunction unsigned short 3
0x013b Artist ascii string
0x013d Predictor unsigned short 1
0x0142 TileWidth unsigned short 1
0x0143 TileLength unsigned short 1
0x0144 TileOffsets unsigned long
0x0145 TileByteCounts unsigned short
0x014a SubIFDs unsigned long
0x015b JPEGTables undefined
0x828d CFARepeatPatternDim unsigned short 2
0x828e CFAPattern unsigned byte
0x828f BatteryLevel unsigned rational 1
0x83bb IPTC/NAA unsigned long
0x8773 InterColorProfile undefined
0x8824 SpectralSensitivity ascii string
0x8825 GPSInfo unsigned long 1
0x8828 OECF undefined
0x8829 Interlace unsigned short 1
0x882a TimeZoneOffset signed short 1
0x882b SelfTimerMode unsigned short 1
0x920b FlashEnergy unsigned rational 1
0x920c SpatialFrequencyResponse undefined
0x920d Noise undefined
0x9211 ImageNumber unsigned long 1
0x9212 SecurityClassification ascii string 1
0x9213 ImageHistory ascii string
0x9214 SubjectLocation unsigned short 4
0x9215 ExposureIndex unsigned rational 1
0x9216 TIFF/EPStandardID unsigned byte 4
0x9290 SubSecTime ascii string
0x9291 SubSecTimeOriginal ascii string
0x9292 SubSecTimeDigitized ascii string
0xa20b FlashEnergy unsigned rational 1
0xa20c SpatialFrequencyResponse unsigned short 1
0xa214 SubjectLocation unsigned short 1
0xa215 ExposureIndex unsigned rational 1
0xa302 CFAPattern undefined 1

Appendix 1: MakerNote of Olympus Digicams

The data below is analyzed at Olympus D450Z(C-920Z) by Peter Esherick.

MakerNote of Olympus Digicam starts from ASCII string "OLYMP". Data format is the same as IFD, it starts from offset 0x07. Example of actual data structure is shown below.

 

Tag No. Tag Name Format CompoNo Value
0x0200 SpecialMode Unsigned Long 3 Shows picture taking mode. First value means 0=normal, 1=unknown, 2=fast, 3=panorama. Second value means sequence number, third value means panorama direction, 1=left to right, 2=right to left, 3=bottom to top, 4=top to bottom.
0x0201 JpegQual Unsigned Short 1 Shows JPEG quality. 1=SQ,2=HQ,3=SHQ.
0x0202 Macro Unsigned Short 1 Shows Macro mode or not. 0=normal, 1=macro.
0x0203 Unknown Unsigned Short 1 Unknown
0x0204 DigiZoom Unsigned Rational 1 Shows Digital Zoom ratio. 0=normal, 2=digital 2x zoom.
0x0205 Unknown Unsigned Rational 1 Unknown
0x0206 Unknown Signed Short 6 Unknown
0x0207 SoftwareRelease Ascii string 5 Shows Firmware version.
0x0208 PictInfo Ascii string 52 Contains ASCII format data such as [PctureInfo]. This is the same data format of older Olympus digicams which not used Exif data format (C1400/C820/D620/D340 etc).
0x0209 CameraID Undefined 32 Contains CameraID data, which is user changeable by some utilities
0x0f00 DataDump Unsigned Long 30 Unknown

 

设置iptables之后不能正常访问ftp解决方法

设置了iptables的禁止所有的端口，只容许可能访问了策略后大部分情况下会出现ftp不能正常访问的问题，因为ftp有主动和被动连接两种模式，少添加一些策略就会出问题。
在这里我就相信的说明下解决方法：
首先加载：
#modprobe ip_conntrack_ftp
#modprobe ip_nat_ftp
然后加载策略
#iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
#iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
#iptables -I INPUT -p tcp --dport 21 -j ACCEPT
#iptables -I OUTPUT -p tcp --dport 21 -j ACCEPT
如果不想每次都加载模块的话请在 /etc/sysconfig/iptables-config
添加

201009231285213452593

保存之后皆可。

Linux上iptables防火墙的基本应用教程

iptables是Linux上常用的防火墙软件，下面给大家说一下iptables的安装、清除iptables规则、iptables只开放指定端口、iptables屏蔽指定ip、ip段及解封、删除已添加的iptables规则等iptables的基本应用。

1、安装iptables防火墙

如果没有安装iptables需要先安装，CentOS执行：
yum install iptables
Debian/Ubuntu执行：
apt-get install iptables

2、清除已有iptables规则

iptables -F
iptables -X
iptables -Z

3、开放指定的端口

#允许本地回环接口(即运行本机访问本机)
iptables -A INPUT -s 127.0.0.1 -d 127.0.0.1 -j ACCEPT
# 允许已建立的或相关连的通行
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
#允许所有本机向外的访问
iptables -A OUTPUT -j ACCEPT
# 允许访问22端口
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
#允许访问80端口
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
#允许FTP服务的21和20端口
iptables -A INPUT -p tcp --dport 21 -j ACCEPT
iptables -A INPUT -p tcp --dport 20 -j ACCEPT
#如果有其他端口的话，规则也类似，稍微修改上述语句就行
#禁止其他未允许的规则访问
iptables -A INPUT -j REJECT （注意：如果22端口未加入允许规则，SSH链接会直接断开。）
iptables -A FORWARD -j REJECT

4、屏蔽IP

#如果只是想屏蔽IP的话“3、开放指定的端口”可以直接跳过。
#屏蔽单个IP的命令是
iptables -I INPUT -s 123.45.6.7 -j DROP
#封整个段即从123.0.0.1到123.255.255.254的命令
iptables -I INPUT -s 123.0.0.0/8 -j DROP
#封IP段即从123.45.0.1到123.45.255.254的命令
iptables -I INPUT -s 124.45.0.0/16 -j DROP
#封IP段即从123.45.6.1到123.45.6.254的命令是
iptables -I INPUT -s 123.45.6.0/24 -j DROP

5、查看已添加的iptables规则

iptables -L -n

v：显示详细信息，包括每条规则的匹配包数量和匹配字节数
x：在 v 的基础上，禁止自动单位换算（K、M）
n：只显示IP地址和端口号，不将ip解析为域名

6、删除已添加的iptables规则

将所有iptables以序号标记显示，执行：
iptables -L -n --line-numbers
比如要删除INPUT里序号为8的规则，执行：
iptables -D INPUT 8

7、iptables的开机启动及规则保存

CentOS上可能会存在安装好iptables后，iptables并不开机自启动，可以执行一下：
chkconfig --level 345 iptables on
将其加入开机启动。
CentOS上可以执行：service iptables save保存规则。
另外更需要注意的是Debian/Ubuntu上iptables是不会保存规则的。
需要按如下步骤进行，让网卡关闭是保存iptables规则，启动时加载iptables规则：
创建/etc/network/if-post-down.d/iptables 文件，添加如下内容：
#!/bin/bash
iptables-save > /etc/iptables.rules
执行：chmod x /etc/network/if-post-down.d/iptables 添加执行权限。
创建/etc/network/if-pre-up.d/iptables 文件，添加如下内容：
#!/bin/bash
iptables-restore < /etc/iptables.rules
执行：chmod x /etc/network/if-pre-up.d/iptables 添加执行权限。
关于更多的iptables的使用方法可以执行：iptables --help或网上搜索一下iptables参数的说明。