Linux target framework (tgt) aims to simplify various SCSI target driver (iSCSI, Fibre Channel, SRP, etc) creation and maintenance. The key goals are the clean integration into the scsi-mid layer and implementing a great portion of tgt in user space.
The developer of IET is also helping to develop Linux SCSI target framework (stgt) which looks like it might lead to an iSCSI target implementation with an upstream kernel component. iSCSI Target can be useful:
a] To setup stateless server / client (used in diskless setups).
b] Share disks and tape drives with remote client over LAN, Wan or the Internet.
c] Setup SAN – Storage array.
d] To setup loadbalanced webcluser using cluster aware Linux file system etc.
In this tutorial you will learn how to have a fully functional Linux iSCSI SAN using tgt framework.
iSCSI target (server)
Storage resource located on an iSCSI server known as a “target”. An iSCSI target usually represents nothing but hard disk storage. As with initiators, software to provide an iSCSI target is available for most mainstream operating systems.
iSCSI initiator (client)
An initiator functions as an iSCSI client. An initiator typically serves the same purpose to a computer as a SCSI bus adapter would, except that instead of physically cabling SCSI devices (like hard drives and tape changers), an iSCSI initiator sends SCSI commands over an IP network.
Debian / Ubuntu Linux Install tgt
Type the following command to install Linux target framework user-space tools:
$ sudo apt-get install tgt
CentOS / RHEL / Red Hat Linux Install tgt
RHEL 5.2 and older version do not have tgt tools. However, RHEL 5.3 (preview version) comes with tgt tools.
tgtadm – Linux SCSI Target Administration Utility
tgtadm is used to monitor and modify everything about Linux SCSI target software: targets, volumes, etc. This tool allows a system to serve block-level SCSI storage to other systems that have a SCSI initiator. This capability is being initially deployed as a Linux iSCSI target, serving storage over a network to any iSCSI initiator.
Start tgtd
To start the tgtd, enter:
# /usr/sbin/tgtd
Under RHEL 5.3 to start the tgtd service, enter:
# /etc/init.d/tgtd start
Define an iscsi target name
The following example creates a target with id 1 (the iqn is 19 iqn.2001-04.com.example:storage.disk2.amiens.sys1.xyz) and adds a 20 logical unit (backed by /dev/hdc1) with lun 1.
# tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2001-04.com.example:storage.disk2.amiens.sys1.xyz
To view the current configuration, enter:
# tgtadm --lld iscsi --op show --mode target
Sample output:
Target 1: iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz
System information:
Driver: iscsi
Status: running
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: deadbeaf1:0
SCSI SN: beaf10
Size: 0
Online: No
Poweron/Reset: Yes
Removable media: No
Backing store: No backing store
Account information:
ACL information:Add a logical unit to the target (/dev/sdb1):
# tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sdb1
A note about home computer / test system
Most production boxes will only use iSCSI root with real iSCSI devices, but for testing purposes it can be quite useful to set up an iSCSI target on your image server. This is useful for testing and learning iSCSI target and iSCSI initiator at home, simply use filesystem for testing purpose. Use dd command to create diskbased filesystem:
# dd if=/dev/zero of=/fs.iscsi.disk bs=1M count=512
Add /fs.iscsi.disk as a logical unit to the target:
# tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /fs.iscsi.disk
Now, you should able to view details:
# tgtadm --lld iscsi --op show --mode target
Sample output:
Target 1: iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz
System information:
Driver: iscsi
Status: running
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: deadbeaf1:0
SCSI SN: beaf10
Size: 0
Online: No
Poweron/Reset: Yes
Removable media: No
Backing store: No backing store
LUN: 1
Type: disk
SCSI ID: deadbeaf1:1
SCSI SN: beaf11
Size: 512M
Online: Yes
Poweron/Reset: Yes
Removable media: No
Backing store: /fs.iscsi.disk
Account information:
ACL information:A Note About Selinux
Renout Gerrits adds – On RHEL and friends you will run into problems if selinux is enabled if using files. While adding a logicalunit to a target you will get the error: ‘tgtadm: invalid request’. A bit of a misguiding error. fixed by setting the correct context:
# semanage fcontext -a -t tgtd_var_lib_t /fs.iscsi.disk
# restorecon -Rv /fs.iscsi.disk
Accept iSCSI Target
To enable the target to accept any initiators, enter:
# tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
This should open network port # 3260:
# netstat -tulpn | grep 3260
Sample output:
tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 27328/tgtd tcp6 0 0 :::3260 :::* LISTEN 27328/tgtd
And you are done. Your system is configured as iSCSI Target. Remote client computer can access this computers hard disk over network. Your can use cluster aware filesystem to setup real shared storage for small business. Open TCP port 3260 in your firewall, if required.
How do I access iSCSI Target (server) via iSCSI initiator (client)?
See detailed os specific iSCSI initiator instuctions:
- RHEL 4 or RHEL 5 Linux iSCSI initiator tutorial.
- Debian Linux iSCSI initiator tutorial.
- FreeBSD iSCSI initiator tutorial.
- Windows iSCSI initiator tutorial.
Following is a quick way to access iSCSI target, under RHEL 5. Let us say your server iSCSI Target IP is 192.168.1.2. Type the following command to discover targets at a given IP address such as 192.168.1.2 (use 127.0.0.1 if you are testing it from same computer):
# iscsiadm --mode discovery --type sendtargets --portal 192.168.1.2
OR
# iscsiadm --mode discovery --type sendtargets --portal 127.0.0.1
Sample output:
127.0.0.1:3260,1 iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz
Login to the iscsi target session:
# iscsiadm --mode node --targetname iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz --portal 192.168.1.2:3260 --login
OR
# iscsiadm --mode node --targetname iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz --portal 127.0.0.1:3260 --login
Verify that login was successful:
# tail -f /var/log/messages
Sample output:
Nov 11 07:34:04 vivek-desktop kernel: [ 9039.562312] scsi 6:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5 Nov 11 07:34:04 vivek-desktop kernel: [ 9039.572268] sd 6:0:0:1: [sdc] 1048576 512-byte hardware sectors (537 MB) Nov 11 07:34:04 vivek-desktop kernel: [ 9039.572374] sd 6:0:0:1: [sdc] Write Protect is off Nov 11 07:34:04 vivek-desktop kernel: [ 9039.572530] sd 6:0:0:1: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Nov 11 07:34:04 vivek-desktop kernel: [ 9039.572715] sd 6:0:0:1: [sdc] 1048576 512-byte hardware sectors (537 MB) Nov 11 07:34:04 vivek-desktop kernel: [ 9039.572790] sd 6:0:0:1: [sdc] Write Protect is off Nov 11 07:34:04 vivek-desktop kernel: [ 9039.572940] sd 6:0:0:1: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Nov 11 07:34:04 vivek-desktop kernel: [ 9039.572946] sdc: unknown partition table Nov 11 07:34:04 vivek-desktop kernel: [ 9039.573492] sd 6:0:0:1: [sdc] Attached SCSI disk Nov 11 07:34:04 vivek-desktop kernel: [ 9039.573593] sd 6:0:0:1: Attached scsi generic sg4 type 0
sdc is new scsi disk. You may need to restart iSCSI to probe partition and check disks:
# service iscsi restart
# partprobe
# fdisk -l
You can now create parition and mount file system using usual fdisk and mkfs.ext3 commands:
# fdisk /dev/sdc
# mkfs.ext3 /dev/sdc1mkdir /iscsi
# mkdir /iscsi
# mount /dev/sdc1 /iscsi
# df -H
Sample output:
Filesystem Size Used Avail Use% Mounted on /dev/sda2 99G 30G 64G 32% / tmpfs 1.1G 0 1.1G 0% /lib/init/rw varrun 1.1G 361k 1.1G 1% /var/run varlock 1.1G 0 1.1G 0% /var/lock udev 1.1G 2.9M 1.1G 1% /dev tmpfs 1.1G 312k 1.1G 1% /dev/shm /dev/sda1 105G 32G 73G 31% /media/sda1 /dev/sda5 294G 275G 20G 94% /share /dev/sdb2 247G 119G 116G 51% /disk1p2 /dev/sdc1 520M 11M 483M 3% /iscsi
References:
- man page tgtadm
- tgt project
- iSCSI article from the wikipedia
30 comment
You guys can try with OPENFILER Opensource SAN Implementation
Britto,
Good suggestion. OPENFILER is really good for SAN.
Nice tutorial. Please post topics on “Cluster aware file system”.
Great how-to, thanks.
tgt is available on Centos 5.2. (details on the introduction to RHEL 5 is at http://rhn.redhat.com/errata/RHEA-2007-0713.html)
To install:
# yum install scsi-target-utils
Works as detailed above.
hi,guys,i have a question,
when i setup a iscsi target then export an iscsi device,
then i discovery the target device on two diffenence initoator ,and mount the device
the in node a i create a file in the device which was iscsi device ,and then i think i will the file in the node b the iscsi device ,but i cann’t see the file .i get lost.
xiaolong
iscsi cannot do file sharing. because, unlike nfs, there is no locking.
just like fc san.
you can make a lun into a zone visible to more than 1 server, you can even access the lun from them. but you do not do that because it is just a raw access. not locking.
hi,
iscsi can export devices, but not filesystems. if you want more than one server access the same filesystem, you have to go on gfs2 or ocfs2. it’s not so difficult to setup 😉
and what about iet vs tgt? what’s the best iscsi target for linux?
many thanks,
ftaurino
amiens.sys1.xyz
is what?
How do you make it persistent after reboot? I am working on ubuntu in my case.
I also would like to know the answer to Vincents question – how to make it survive a reboot
Great post – thanks!
Vincent
For making the setup persistent over reboot type this command:
# tgt-admin –dump > /etc/tgt/targets.conf
# chkconfig tgtd on
How would you make a boot LUN if all you can set it is LUN 1? Or is there anyway to make it bootable? The only OS I’ve been able to boot from that LUN is SLES 11 SP1. Tried RHEL 5 and Win2k8 and both could not locate the LUN.
Thanks for a good tutorial. First one I found that was actually helpful in getting everything setup, bookmarked.
@John
If you are booting Win2k8 “from SAN” per se, the boot partition has to have a LUN # that is neither 0 or 1 and is not more than 3 digits (ex. LUN 100, 101).
I’m on CentOS 5.4 and when i run this command getting below error:
tgtadm –lld iscsi –op new –mode target –tid 1 -T iqn.2001-04.com.example:storage.disk2.amiens.sys1.xyz
tgtadm: unrecognized option ‘iscsi’
try ‘tgtadm –help’
can you please help? Thanks
At first glance it sounds like something isn’t correct with the iscsi utilities, I’d try reinstalling them.
yum reinstall iscsi-initiator-utils scsi-target-utils
you should add the sudo command as
sudo tgtadm –lld iscsi –op new –mode target –tid 1 -T iqn.2001-04.com.example:storage.disk2.amiens.sys1.xyz
Thanks for the quick response. I was missing “iscsi-initiator-utils”. After installing it “tgtadm –lld iscsi –op show –mode target” started showing results. But is not persistent after reboot, I did try steps from above comment(# tgt-admin –dump > /etc/tgt/targets.conf and # chkconfig tgtd on) and it still doesn’t work. I’m using CentOS 5.4. Thanks again for the wonderful post.
Never mind. I re-did the configuration from scratch and now it is persistent after reboots. Thanks again
is it possible to do high available cluster environment using this technology??
It is so long as you remember to use a locking filesystem properly. Something like drbl. Can’t say I’ve ever set one up so that’s all for now.
It’s really nice tutorial. I am doing a LAB setup. Where I am using Open Filer for storage. In 2 node RHEL5 systems I need to share the same LUN. using “iscsiadm –mode discovery –type sendtargets –portal 192.168.1.2” I am able to discover the LUN in both of the systems.
I have implement one iscsi target . Now wont to implement two iscsi targets with load balancing. is it posible to do so?…..
is it is …
you nee to use mdadm or LVM mirroring , discover the 2 devices then set up a single device to mount , if you are using a high storage system with ISCSI capabilites then you can use multipath to bind the 2 devices.
before that you need to make sure the initiator have 2 different IP’s (of the same storage ) to talk with.
Hello,
Is it possible to connect 2 initiators on the same target (initiators and target on centos 6)?, each initiator accessing different block device of the target?
I have tried to configured the /etc/targets.conf using initiator-address ,and also using incominguser password, but it seems these are global variable and can not be defined per lun. Any ideas?
I have a rhel6.1 setup and the below command is not working.. is it a BUG.
/usr/sbin/tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 1 -b /mnt1/lun10
return code 22
==============================================
tgtadm: invalid request
==============================================
faced
NO Portal Found Error
Fixed the issue using your Command
Thousands of Thanks …
When I issue the command “/usr/sbin/tgtd” I get the following:
(null): iscsi_tcp_init_portal(227) unable to bind server socket, Address already in use
(null): iscsi_tcp_init_portal(227) unable to bind server socket, Address already in use
(null): iscsi_add_portal(275) failed to create/bind to portal (null):3260
librdmacm: couldn’t read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
(null): iser_ib_init(3263) Failed to initialize RDMA; load kernel modules?
(null): fcoe_init(214) (null)
(null): fcoe_create_interface(171) no interface specified.
(null): ipc_init(635) can’t bind a socket, Address already in use
Any idea what I can do?
Running Linux Mint Maya (13) Kernel 3.2.0-23-generic
Thanks
Hello,
I did the following procedure:
tgtadm –lld iscsi –mode target –op new –tid 1 –targetname iqn.2014-09.local.vclass:tg0
tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 1 –backing-store /dev/sdg
tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 2 –backing-store /dev/sdh
tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 3 –backing-store /dev/sdi
tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 4 –backing-store /dev/sdj
tgtadm –lld iscsi –mode target –op new –tid 2 –targetname iqn.2014-09.local.vclass:tg1
tgtadm –lld iscsi –mode target –op new –tid 3 –targetname iqn.2014-09.local.vclass:tg2
tgtadm –lld iscsi –op new –mode logicalunit –tid 2 –lun 5 –backing-store /dev/sde
tgtadm –lld iscsi –op new –mode logicalunit –tid 3 –lun 5 –backing-store /dev/sdf
tgtadm –lld iscsi –op bind –mode target –tid 2 –initiator-address 172.20.10.61
tgtadm –lld iscsi –op bind –mode target –tid 3 –initiator-address 172.20.10.62
tgtadm –mode target –op show
tgt-admin –dump > /etc/tgt/targets.conf
and the tgtadm –mode target –op show output was as shown bellow:
[[email protected] ~]# tgtadm –mode target –op show
Target 1: iqn.2014-09.local.vclass:tg0
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 10737 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sdg
Backing store flags:
LUN: 2
Type: disk
SCSI ID: IET 00010002
SCSI SN: beaf12
Size: 10737 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sdh
Backing store flags:
LUN: 3
Type: disk
SCSI ID: IET 00010003
SCSI SN: beaf13
Size: 10737 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sdi
Backing store flags:
LUN: 4
Type: disk
SCSI ID: IET 00010004
SCSI SN: beaf14
Size: 10737 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sdj
Backing store flags:
Account information:
ACL information:
Target 2: iqn.2014-09.local.vclass:tg1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00020000
SCSI SN: beaf20
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 5
Type: disk
SCSI ID: IET 00020005
SCSI SN: beaf25
Size: 10737 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sde
Backing store flags:
Account information:
ACL information:
172.20.10.61
Target 3: iqn.2014-09.local.vclass:tg2
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00030000
SCSI SN: beaf30
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 5
Type: disk
SCSI ID: IET 00030005
SCSI SN: beaf35
Size: 10737 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sdf
Backing store flags:
Account information:
ACL information:
172.20.10.62
My intention is that all LUN IDs should be different per host, so tg0 will serves shared LUNs 1,2,3 and 4 and tg1 serves LUN 5 to host 172.20.10.61 and tg2 serves LUN 5 to host 172.20.10.62. I am using CentOS 6.5
After reboot tg1 and tg2 presents the LUNs both with LUN ID 1.
Also added LUN 5 to both targets in /etc/tgt/targets.conf and after reboot ignores the instruction and still presents with LUN ID 1.
/etc/tgt/targets.conf bellow:
Default-driver iscsi
backing-store /dev/sdf
LUN 5
initiator-address 172.20.10.62
backing-store /dev/sde
LUN 5
initiator-address 172.20.10.61
backing-store /dev/sdg
backing-store /dev/sdh
backing-store /dev/sdi
backing-store /dev/sdj
Please, can anyone be so kind and help me to fix the LUN IDs?
Thank you in advance.
Hi,
Using Centos nodes & Openfiler. Quorum disk shows up as sdb on node1 and the quorum disk is sdc on node2. Similarly shared-disk is sdc on node1 and sdb on node2. Getting a couple of disks for “quorum” disk and “shared disk” i.e a total of 4 disks on 1 node each…maybe the 2 interfaces which I am using on the virtual-box is giving this o/p !!!!.I have tried rediscovering the targets(logout,delete,discover & relogin) and reloading the iscsi service as well. Is there any way to fix this.